概述
本次项目，采用FastDFS文件系统，存储头像文件。
FastDFS
分布式⽂件系统（Distributed File System）是指⽂件系统管理的物理存储资源不⼀定直接连接在本地节点上，⽽是通过计算机⽹络与节点相连。
判断⼀个分布式⽂件系统是否优秀，取决于以下三个因素：
①.数据的存储⽅式
②/数据的读取速率
③.数据的安全机制
FastDFS 是⼀款开源的轻量级分布式⽂件系统如下特点：
- 纯粹C语⾔实现，⽀持Linux、FreeBSD等unix系统。
- 类似GoogleFS/HDFS，但是不是通⽤的⽂件系统，只能通过专有的API访问，⽬前提供了C、Java和PHPAPI 互联⽹量身定做，追求⾼性能，⾼扩展.
- FastDFS不仅仅可以存储⽂件，还可以存储⽂件的元数据信息（可选）。
架构：
整个FastDFS架构中有Client、Tracker和Storage服务。
其中Client⽤于提交⽂件给FastDFS集群。
Storage 服务负责实际数据的存储。
Tracker服务负责监控和调度Storage服务，起到负载均衡器的作⽤。
安装
Github链接：https://github.com/happyfish100 安装包获取：https://pan.baidu.com/s/1k3v7j826qlYIHwO3Q84FSg 获取码：j68s
- 以3台服务器为样例：所有带 x的是指所有服务器都要安装。
①.安装依赖包
[root@hbaseX ~] yum install -y gcc-c++ #下载c++依赖
[root@hbaseX ~] tar -zxf libfastcommon-1.0.39.tar.gz
[root@hbaseX ~] cd libfastcommon-1.0.39
[root@hbaseX libfastcommon-1.0.39] ./make.sh
[root@hbaseX libfastcommon-1.0.39] ./make.sh install②安装FastDFS
[root@hbaseX ~] yum install -y perl-devel #下载依赖插件
[root@hbaseX ~] tar -zxf fastdfs-5.11.tar.gz
[root@hbaseX ~] cd fastdfs-5.11
[root@hbaseX fastdfs-5.11] ./make.sh
[root@hbaseX fastdfs-5.11] ./make.sh install③配置
创建fdfs运行所需的数据目录
[root@hbaseX ~] mkdir -p /data/fdfs/{tracker,storage/store01,storage/store02}创建启动所需的配置文件
[root@hbaseX ~] cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf
[root@hbaseX ~] cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
[root@hbaseX ~] cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf配置Tracker Server
[root@hbaseX ~] vi /etc/fdfs/tracker.conf修改以下配置：
base_path=/data/fdfs/tracker配置Storage Server
[root@hbaseX ~] vi /etc/fdfs/storage.conf修改以下配置：
group_name=group[1,2,3] #这里的1,2，3，指代分别在美个服务器创建不同的组名
base_path=/data/fdfs/storage
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02
tracker_server=hbase1:22122
tracker_server=hbase2:22122
tracker_server=hbase3:22122修改client
[root@hbaseX ~] vi /etc/fdfs/client.conf修改以下配置：
base_path=/tmp
tracker_server=hbase1:22122
tracker_server=hbase2:22122
tracker_server=hbase3:22122④启动
保证所有节点都首先启动 tracker
#启动追踪器，一定所有服务器都先启动
[root@hbaseX ~] /etc/init.d/fdfs_trackerd start
Starting fdfs_trackerd (via systemctl): [ 确定 ]
[root@hbaseX ~] /etc/init.d/fdfs_storaged start
Starting fdfs_storaged (via systemctl): [ 确定 ]
[root@hbaseX ~] ps -aux | grep fdfs
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 78950 0.0 0.1 144784 2040 ? Sl 21:06 0:00 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 79000 13.0 3.2 83520 67144 ? Sl 21:06 0:06 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf
root 79324 0.0 0.0 103320 884 pts/0 S+ 21:07 0:00 grep fdfs⑤命令测试
上传文件
[root@hbase1 ~] fdfs_upload_file /etc/fdfs/client.conf iviolet_evergarden-009.jpg
#返回的文件路径
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.jpg查看文件信息
[root@CentOSX ~] fdfs_file_info /etc/fdfs/client.conf
#返回的信息
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.jpg
source storage id: 0
source ip address: 192.168.164.129
file create timestamp: 2019-06-30 21:11:41
file size: 8901
file crc32: 2662613490 (0x9EB441F2)下载文件
[root@hbase2 ~] fdfs_download_file /etc/fdfs/client.conf group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.jpg删除文件
[root@hbase3~] fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.jpg追加文件信息
[root@hbase2 ~] echo "hello" > 1.txt
[root@hbase2 ~] echo "word" > 2.txt
[root@hbase2 ~] fdfs_upload_appender /etc/fdfs/client.conf /root/1.txt
#返回的文件信息
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
[root@hbase2 ~] fdfs_append_file /etc/fdfs/client.conf group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt 2.txt
[root@hbase2 ~] fdfs_download_file /etc/fdfs/client.conf group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
[root@hbase2 ~] cat wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
hello
word监视状态
[root@hbase2 ~] fdfs_monitor /etc/fdfs/client.conf文件校验和
[root@hbase2 ~] fdfs_crc32 /etc/fdfs/client.conf group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
#返回的值
2911662598Nginx集成FastDFS
①安装
[root@hbaseX ~] tar -zxf fastdfs-nginx-module.tar.gz
[root@hbaseX ~] yum install -y pcre-devel
[root@hbaseX ~] yum install -y openssl-devel
[root@hbaseX ~] tar -zxf nginx-1.11.1.tar.gz
[root@hbaseX ~] cd nginx-1.11.1/
[root@hbaseX nginx-1.11.1] ./configure --prefix=/usr/local/nginx-1.11.1/ --add-module=/root/fastdfs-nginx-module/src
[root@hbaseX nginx-1.11.1] make
[root@hbaseX nginx-1.11.1] make install②配置
[root@hbaseX ~] cp fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
[root@hbaseX ~] cd fastdfs-5.11/conf/
[root@hbaseX conf] cp http.conf mime.types anti-steal.jpg /etc/fdfs/nginx配置
root@hbaseX ~] vi /usr/local/nginx-1.11.1/conf/nginx.conf添加以下配置
#access_log logs/host.access.log main;
location ~ /group[0-9]+/M00 {
root /data/fdfs/storage/store01;
ngx_fastdfs_module;
}
location ~ /group[0-9]+/M01 {
root /data/fdfs/storage/store02;
ngx_fastdfs_module;
}
location / {
root html;
index index.html index.htm;
}
#error_page 404 /404.html;配置mod_fastdfs.conf
[root@hbaseX ~] vi /etc/fdfs/mod_fastdfs.conf添加以下配置
tracker_server=hbase1:22122
tracker_server=hbase2:22122
tracker_server=hbase3:22122
group_name=group[1,2,3] #这里的1,2，3，指代分别在美个服务器创建不同的组名
url_have_group_name = true
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02③启动
[root@hbaseX ~] cd /usr/local/nginx-1.11.1/
[root@hbaseX nginx-1.11.1] ./sbin/nginx -t
# 检查语法是否正确
ngx_http_fastdfs_set pid=116305
ngx_http_fastdfs_set pid=116305
nginx: the configuration file /usr/local/nginx-1.11.1//conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-1.11.1//conf/nginx.conf test is successful
#启动
[root@hbaseX nginx-1.11.1] ./sbin/nginx④访问测试
[root@hbaseX ~] fdfs_upload_file /etc/fdfs/client.conf 2020-03-20-22-52-35_0.png
#返回的文件位置
group2/M00/00/00/CgoAmF520H6APRvGACUXSBEcDIk006.png浏览器地址访问：ip地址/group2/M00/00/00/CgoAmF520H6APRvGACUXSBEcDIk006.png
需要下载时：ip地址/group2/M00/00/00/CgoAmF520H6APRvGACUXSBEcDIk006.png?filename=自定义文件名.png
FastDHT去重
①安装BerkeleyDB
[root@hbaseX ~] tar -zxf db-4.7.25.tar.gz
[root@hbaseX ~] cd db-4.7.25
[root@hbaseX db-4.7.25] cd build_unix/
[root@hbaseX build_unix] ./../dist/configure
[root@hbaseX build_unix] make
[root@hbaseX build_unix] make install②安装FastDHT
[root@hbaseX ~] tar zxf FastDHT_v2.01.tar.gz
[root@hbaseX ~] cd FastDHT
[root@hbaseX FastDHT] ./make.sh
[root@hbaseX FastDHT] ./make.sh install③配置
修改fdhtd.conf
[root@hbaseX ~] mkdir /data/fastdht
[root@hbaseX ~] vi /etc/fdht/fdhtd.conf修改以下配置：
base_path=/data/fastdht修改fdht_servers.conf
[root@hbaseX ~] vi /etc/fdht/fdht_servers.conf修改以下配置：
group_count = 3
group0 = hbase1 :11411
group1 = hbase2 :11411
group2 = hbase3 :11411修改fdht_client.conf配置文件
[root@hbaseX ~] vi /etc/fdht/fdht_client.conf修改以下配置：
base_path=/tmp/④启动
[root@hbaseX ~] fdhtd /etc/fdht/fdhtd.conf start⑤测试
[root@hbaseX ~] ps -axu| grep fd
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 29127 0.0 0.8 195200 17504 ? Sl 00:38 0:00 fdhtd /etc/fdht/fdhtd.conf start
root 29193 0.0 0.0 103320 884 pts/0 S+ 00:39 0:00 grep fd
root 128672 0.2 0.1 275856 2204 ? Sl 00:14 0:03 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 128710 1.8 3.2 85584 67188 ? Sl 00:14 0:27 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf命令测试
储存
[root@hbaseX ~] fdht_set /etc/fdht/fdht_client.conf test1:user001 name='zs',age=18;读取
[root@hbaseX ~] fdht_get /etc/fdht/fdht_client.conf test1:user001 name,age删除
[root@hbaseX ~] fdht_delete /etc/fdht/fdht_client.conf test1:user001 name;⑥集成FastDHT
修改etc/fdfs/storage.conf配置文件
[root@hbaseX ~] vi /etc/fdfs/storage.conf修改以下配置：
check_file_duplicate=1
keep_alive=1
#此处为C语言的头文件导入
#include /etc/fdht/fdht_servers.conf分别启动fdhtd服务、fastfs
[root@hbaseX usr] /usr/local/bin/fdhtd /etc/fdht/fdhtd.conf restart
[root@hbaseX usr] /etc/init.d/fdfs_trackerd restart
[root@hbaseX usr] /etc/init.d/fdfs_storaged restart集成SpringBoot
①依赖
<!--FastDFS系统-->
<dependency>
<groupId>com.github.tobato</groupId>
<artifactId>fastdfs-client</artifactId>
<version>1.26.6</version>
</dependency>②yml文件配置
#文件系统
fdfs:
tracker-list: pro1:22122,pro2:22122,pro3:2122
# 配置默认缩略图
thumb-image:
height: 80
width: 100③上传头像业务集成
package com.baizhi.controller;
import com.baizhi.entities.User;
import com.baizhi.service.IUserService;
import com.github.tobato.fastdfs.domain.fdfs.StorePath;
import com.github.tobato.fastdfs.domain.upload.FastImageFile;
import com.github.tobato.fastdfs.domain.upload.ThumbImage;
import com.github.tobato.fastdfs.service.FastFileStorageClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import java.io.IOException;
import java.util.HashSet;
import java.util.List;
@RestController
@RequestMapping(value = "/restUserManager")
public class RestUserController {
//注入一个文件存储客户端
@Autowired
private FastFileStorageClient fastFileStorageClient;
//用户注册---增
@PostMapping(value = "/registerUser")
public User registerUser(@RequestPart(value = "user")User user,
@RequestParam(value = "multipartFile",required = false) MultipartFile multipartFile) throws IOException{
//定义文件名
String photo = null;
//如果头像图片不为空
if (multipartFile!=null) {
//获取文件原始名
// String filename = multipartFile.getOriginalFilename();
//获取文件后缀名 //FilenameUtils.getExtension(filename)
//获取文件的类型,如果是图片，则继续进行，否则，返回。
if(!("image".equals(multipartFile.getContentType().split("/")[0]))){
return null;
}else {
//获取文件的输入流，存入文件系统
FastImageFile fastImageFile = new FastImageFile(multipartFile.getInputStream(), multipartFile.getSize(), multipartFile.getContentType().split("/")[1], new HashSet<>(), new ThumbImage(100, 80));
//上传文件
StorePath storePath = fastFileStorageClient.uploadImage(fastImageFile);
//设置真实文件路径，存入数据库
photo = storePath.getFullPath();
}
}
//储存用户信息
user.setPhoto(photo);
//调用接口
try{
//调用接口
iUserService.saveUser(user);
}catch (Exception e){
fastFileStorageClient.deleteFile(photo);
}
return user;
}
//删除
@DeleteMapping(value = "/deleteUserByIds")
public void delteUserByIds(@RequestParam(value = "ids") Integer[] ids){
//先查询此用户的头像文件名
for (Integer id : ids) {
User user = iUserService.queryUserById(id);
fastFileStorageClient.deleteFile(user.getPhoto());
}
//调用接口
iUserService.deleteByUserIds(ids);
}
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
