前言
从事大疆行业应用开发有一段时间了，看到很多厂商在做视频回传的时候，都要装个自己的APP，界面很丑不说，还经常卡死，但是大疆其实已经在视频流中携带了很多信息，很多人都不知道，现在把自己的直播开发经验分享出来
文章目录
- 前言
- 一、安防视频平台介绍
- 二、DJI Pilot机型匹配表及回传流程
- 1. 视频流选择
- 2. 视频回传的码流格式
- 3. GB28181业务流程
- 4. 传输和控制要求（基于 GB/T 28181）
- 5. 功能列表
- 三、DJI Pilot H264 SEI帧 Metadata协议解析
- 1. Metadata 数据列表
- 2. DJI Metadata protobuf解析文件
- 3. Metadata 解析指导
- 四、DJI GB28181 协议信令详解
- 总结
一、安防视频平台介绍
二、DJI Pilot机型匹配表及回传流程
APP 版本 | DJI Pilot V1.9 及以上版本 | DJI Pilot PE V1.6.1 |
适配机型 | 经纬 M300 RTK 经纬 M200 V2 系列 御 Mavic 2 行业变焦版 御 Mavic 2 行业双光版 精灵 Phantom 4 RTK（SDK 遥控器版本） | 经纬 M200 V1 系列 经纬 M600 系列 悟 Inspire 2 御 Mavic Pro 精灵 Phantom 4 精灵 Phantom 4 Pro（ 除精灵 4 Pro+） 精灵 Phantom 4 Advanced（除精灵 4 Advanced+） |
Pilot APP 下载：https://www.dji.com/cn/downloads/djiapp/dji-pilot Pilot PE V1.6.1 下载：https://service-adhoc.dji.com/download/app/android/321b585b-e217-4c95-9192-09eee6e02630
1. 视频流选择
M300RTK 最多可以挂载 3 个相机负载，可以通过遥控器上的 APP 从三路相机视频流
中选择某一路视频流进行回传到服务端。
- FPV 视频流：无人机前置摄像头第一视角视频流。
- 1 号/2 号/3 号负载相机视频流：无人机上的 3 个云台口，可以任意挂载大疆的官方负载 H20T/ H20/ Z30/ XTS 相机，以及三方合作伙伴的相机。
- H20T/H20 相机负载：多光组合相机，具备 1200 万像素广角、2000 万像素 23 倍光学变焦、1200 米激光测距、红外（H20T 有）。
- Z30 相机负载：30 倍光学变焦相机。
- XTS 相机负载：红外相机。
2. 视频回传的码流格式
- 编码格式：H264。
- 质量调节：流畅 540P 30FPS 512Kbps， 均衡 720P 30FPS 1Mbps， 高清 720P 30FPS 1.5 Mbps。
- 传输格式：根据 GB/T28181-2016 传输要求,采用基于 RTP 的视频数据 PS 封装。
3. GB28181业务流程
4. 传输和控制要求（基于 GB/T 28181）
- SIP 信令通信采用 UDP 协议，端口自定义，默认 5060。
- RTP 数据通信采用 UDP 协议，多路流分别采用不同的端口分别传输，点播时通过SDP 协商，端口自定义。
- RTCP 传输控制协议可选，默认 RTP 端口号加 1，需要收发双方同时支持才可，目前仅适用于网络质量反馈，暂不支持丢包重传机制。
- 无人机视频编码格式为 H264，封装成 PS 流，再用 RTP 打包传输；不支持音频。
- 接入网络应为 WiFi/有线/4G/VPDN 的一种，并在现场为遥控器提供 WiFi 热点。
- 遥控器到服务器端的网络带宽应满足以下标准：流畅 512Kbps，均衡 1Mbps，高清1.5Mbps。
- 接收端应具备 jitterbuffer 接收缓冲，以消除网络抖动和乱序造成的视频卡顿和花屏 ，建议缓冲不要超过 1 秒。
5. 功能列表
功能点 | 功能描述 | 协议接口 | 备注 |
设备注册注销 | 采用基于账号口令认证的设备登录与注销；采用定时心跳保活机制，在心跳超时离线后，自动重登录。 | ||
设备信息查询 | 支持远程查询无人机名称，ID，无人机型号，生产厂商，固件版本等信息 | ||
设备状态查询 | 支持远程查询无人机工作状态，视频流状态，录像状态，UTC时间等。 | ||
设备列表查询 | 支持远程查询摄像头列表，即具备多通道视频传输的能力。 | ||
实时视频点播与停止 | 设备上线后，采用点播即传的策略，按需上传，节省通信带宽和流量。 | ||
设备控制强制发送关键帧 | 当视频流丢帧或切换分辨率时，可通过设备控制信令，强制无人机立即编码发送一个 H264 关键帧，减少播放等待时间。 |
三、DJI Pilot H264 SEI帧 Metadata协议解析
目前无人机直播所用的 RTMP 协议只支持标准 RTMP 视频流及其内置的媒体控制命令，没有其他定制化
功能。
- 编码格式：H.264，有 SEI 帧（SEI 帧主要是用于存放 Metadta 信息）
- 传输格式：采用 RTMP 的 chunk 方式进行传输
- 码流质量调节：
- 不支持服务器进行质量设置调节。
- 支持 APP 手动设置，参数如下：
- 流畅 540P 30FPS 512Kbps
- 均衡 720P 30FPS 1Mbps
- 高清 720P 30FPS 1.5Mbps
- 默认端口：默认使用 1935 端口，若服务器指定其他端口的话，需要先 PING 测试。
- Metadata 数据项：
- 飞机经纬高
- 遥控器经纬高
- 目标点位置信息
1. Metadata 数据列表
根据大疆内部文件列出，经实测有改动，具体以proto文件为准
类别 | Metadata 名称 | 类型 | 单位 | 推送频率 |
飞机信息 | 飞机位置-经度 | double | 度 | 10Hz |
飞机信息 | 飞机位置-纬度 | double | 度 | 10Hz |
飞机信息 | 飞机位置-椭球高 | float | 米 | 10Hz |
飞机信息 | 飞机姿态-roll | Int32 | 0.1 度 | 10Hz |
飞机信息 | 飞机姿态-yaw | Int32 | 0.1 度 | 10Hz |
飞机信息 | 飞机姿态-pitch | Int32 | 0.1 度 | 10Hz |
飞机信息 | Home 点-经度 | Double | 度 | 1Hz |
飞机信息 | Home 点-纬度 | Double | 度 | 1Hz |
飞机信息 | Home 点-椭球高 | float | 米 | 1Hz |
遥控器信息 | 遥控器角色 | enum | 0.2Hz | |
遥控器信息 | 遥控器位置-经度 | double | 度 | 0.2Hz |
遥控器信息 | 遥控器位置-纬度 | double | 度 | 0.2Hz |
遥控器信息 | 遥控器位置-椭球高 | float | 米 | 0.2Hz |
目标点 | TargetPoint-版本 | Uint32 | 2Hz | |
目标点 | TargetPoint-Source | Char * | 2Hz | |
目标点 | TargetPoint-数据[数组] | 2Hz |
TargetPoint-数据[数组]
参数名 | 类型 | 单位 |
type | Enum | |
index | Uint32 | |
latitude | Double | 度 |
longitude | Double | 度 |
altitude_ellipsoid | Float | 米 |
altitude_relative | Float | 米 |
video_stream_window_x | Float | |
video_stream_window_y | Float | |
flags | Uint32 |
2. DJI Metadata protobuf解析文件
DJI对Metadata采用protobuf进行封装，以下是proto文件，已经本人验证并修改了部分BUG
dvtm_library.proto
// DJI Video Timed Metadata Format using Protobuf 3
// version = 3
syntax = "proto3";
// basic message defin ==========================
enum DIRECTION_TYPE {
RESERVED = 0;
NORTH = 1;
NORTHEAST = 2;
EAST = 3;
SOUTHEAST = 4;
SOUTH = 5;
SOUTHWEST = 6;
WEST = 7;
NORTHWEST = 8;
}
message DjiUTC {
uint32 year = 1; // The year, like 2020.
uint32 month = 2; // The month in the year, in the range 1 to 12.
uint32 day = 3; // The day of the month, in the range 1 to 31.
uint32 hour = 4; // The number of hours past midnight, in the range 0 to 23.
uint32 min = 5; // The number of minutes after the hour, in the range 0 to 59.
uint32 sec = 6; // The number of seconds after the minute, normally in the range 0 to 59.
uint32 nsec = 7; // The number of nanoseconds after the second.
}
message DeviceIdentify {
string serial_number = 1; // limited to 32 characters
}
message DjiInherentInfoBase {
DeviceIdentify device_identify = 1; // Include sn now.
string module_name = 2; // limited to 32 characters
string firmware_version = 3; // limited to 32 characters
string manufacture_id = 4; // limited to 18 characters
string product_type = 5; // limited to 32 characters
}
// [end] basic message defin ==========================
// Global Information, only appear once in whole video ==========================
message DjiModuleInfo {
enum ModuleType {
// The body of any kinds of camera which generate this video,
// including gimbal camera, payload camera, or fly camera with
// non-removable drone.
MODULE_CAMERA_BODY = 0;
// interchangeable drone
MODULE_DRONE = 1;
// interchangeable lens
MODULE_LENS = 2;
}
ModuleType module_type = 1;
string model_name = 2; // limited to 32 characters
string serial_number = 3; // limited to 32 characters
string firmware_version = 4; // limited to 32 characters
}
message DjiVideoGlobalInfo {
repeated DjiModuleInfo module_info = 1;
// The video full path file name, limited to 800 characters
string file_name = 3;
fixed32 video_uuid = 4; // the video unique ID
string record_start_time = 5; // uses RFC 3339
uint32 resolution_height = 6; // unit: pixel
uint32 resolution_width = 7; // unit: pixel
float video_framerate = 8;
enum VideoType {
VIDEO_NORMAL = 0;
VIDEO_DELAY = 1;
VIDEO_SLOW_MOTION = 2;
VIDEO_QUICK_MOVIE = 3;
VIDEO_TIMELAPSE = 4;
VIDEO_MOTIONLAPSE = 5;
VIDEO_HYPERLAPSE = 6;
VIDEO_HDR = 7;
VIDEO_LOOP_RECORD = 8;
}
VideoType video_type = 9;
enum VideoEncoder {
ENCODER_H264 = 0;
ENCODER_H265 = 1;
}
VideoEncoder video_encoder = 10;
uint32 library_proto_version = 11; // version of dvtm_library.proto
uint32 product_proto_version = 12; // version of product corresponding proto file
}
// [end] Global Information, only appear once in whole video ====================
// Describe every data source as a message ===================================
// All device_id field in messages is internal reserved at present.
// User should not care about or depends on this field.
message DjiCameraBasic {
uint32 device_id = 1;
int64 timestamp = 2;
uint32 frame_id = 3;
// reserved for future use
string camera_name = 4; // limited to 32 characters
sint32 exposure_bias_tenfold = 5; // exposure bias, tenfold expression, reserved for IR camera
float exposure_time = 6; // exposure time (uint: s)
uint32 iso = 7; // photographic sensitivity, reserved for IR camera
uint32 fnumber_tenfold = 8; // F-Number, tenfold expression
float focal_length = 9; // actual focal length in mm
float digital_zoom_ratio = 10;
}
message DjiLaserRanging {
uint32 device_id = 1;
int64 timestamp = 2;
uint32 frame_id = 3;
// processed data of laser ranging function, fields 5-10 is only valid
// if ranging_enabled is true, which means you enable this function
// by yourself on APP. fields 6-8 is only valid if gps_status in DjiGpsBasic
// message is not GPS_INVALID.
bool ranging_enabled = 4;
uint32 target_distance = 5; // mm
float target_longitude = 6; // degree
float target_latitude = 7; // degree
uint32 target_altitude = 8; // height relative to takeoff point in mm
uint32 screen_offset_x = 9; // target offset on horizontal direction of screen in permillage
uint32 screen_offset_y = 10; // target offset on vertical direction of screen in permillage
// raw sensor data, always on. fields 12-19 is only valid if laser_status is LASER_NORMAL.
enum LaserStatus {
LASER_NORMAL = 0; // laser ranging finder works fine.
LASER_TOO_CLOSE = 1; // target distance is less than minimum range of finder.
LASER_TOO_FAR = 2; // target distance is larger than maximum range of finder.
LASER_CLOSED = 3; // laser module is closed.
}
LaserStatus laser_status = 11;
uint32 distance1 = 12; // unit: mm
uint32 intensity1 = 13; // signal intensity, range: 0~255
uint32 distance2 = 14; // unit: mm
uint32 intensity2 = 15; // signal intensity, range: 0~255
uint32 distance3 = 16; // unit: mm
uint32 intensity3 = 17; // signal intensity, range: 0~255
uint32 distance4 = 18; // unit: mm
uint32 intensity4 = 19; // signal intensity, range: 0~255
uint32 target_abs_alt = 20; // absolute height of target point in mm
}
message DjiGpsBasic {
uint32 device_id = 1;
int64 timestamp = 2;
uint32 frame_id = 3;
double gps_latitude = 4; // unit: rad, WGS-84 coordinate system
double gps_longitude = 5; // unit: rad, WGS-84 coordinate system
int32 gps_altitude_mm = 6; // unit mm, refer to gps_altitude_type for details
enum GpsStatus {
GPS_NORMAL = 0; // working with non-RTK GPS
GPS_INVALID = 1; // GPS signal is non-available, measurement interrupted
GPS_RTK = 2; // working with RTK-GPS
}
GpsStatus gps_status = 7;
// Specify type of gps_altitude_mm and altitudes in other messages. For versions that don't support this fields,
// use gps_status to get the type. If gps_status is GPS_RTK, gps_altitude_type is RTK_ALTITUDE, otherwise,
// gps_altitude_type is PRESSURE_ALTITUDE.
enum GpsAltType {
PRESSURE_ALTITUDE = 0; // altitude is mainly provided by barometer which has different origin with ellipsoidal height
GPS_FUSION_ALTITUDE = 1; // fused height by GPS and barometer, which based on ellipsoidal coordinate
RTK_ALTITUDE = 2; // altitude is ellipsoidal height provided by RTK
}
GpsAltType gps_altitude_type = 8;
enum COORDINATE_TYPE {
WGS84 = 0;
CGCS2000 = 1;
}
COORDINATE_TYPE coordinate = 9; // In rtk or gps mode, it indicate the coordinate of the lat, lon, alti.
DjiUTC utc_time = 10;
DeviceIdentify device_identify = 11;
}
message DjiFlyingState {
uint32 device_id = 1;
int64 timestamp = 2;
uint32 frame_id = 3;
// drone body speed components in NED coordinate system.
int32 speed_x_dms = 4; // north direction, unit: 0.1 m/s
int32 speed_y_dms = 5; // east direction, unit: 0.1 m/s
int32 speed_z_dms = 6; // vertical direction, unit: 0.1 m/s
// The Euler angles of drone body relative to the NED (North, East, Down)
// coordinate system. Rotation sequence of the Euler angle is
// ZYX (yaw, pitch, roll), intrinsic.
sint32 pitch_decidegree = 7; // unit: 0.1 degree
sint32 roll_decidegree = 8; // unit: 0.1 degree
sint32 yaw_decidegree = 9; // unit: 0.1 degree
// height relative to home point, unit: 0.1m
sint32 relative_height_decimeter = 10;
DjiUTC utc_time = 11;
int32 heading = 12; // unit: 0.1 degree
}
message DjiGimbal {
uint32 device_id = 1;
int64 timestamp = 2;
uint32 frame_id = 3;
enum GimbalPosition {
GIMBAL_POS_NORMAL = 0;
GIMBAL_POS_REVERSE = 1;
}
GimbalPosition gimbal_position = 4;
enum GimbalMode {
GIMBAL_MODE_FREE = 0;
GIMBAL_MODE_FPV = 1;
GIMBAL_MODE_FOLLOW = 2;
}
GimbalMode gimbal_mode = 5;
// The Euler angles of gimbal relative to the NED (North, East, Down)
// coordinate system. Rotation sequence of the Euler angle is
// ZXY (yaw, roll, pitch), intrinsic. For upward gimbal, the Euler
// angles translate from the real quaternion of gimbal after rotate
// 180 degree around the X axis of moving body.
sint32 pitch_decidegree = 6; // unit: 0.1 degree
sint32 roll_decidegree = 7; // unit: 0.1 degree
sint32 yaw_decidegree = 8; // unit: 0.1 degree
DjiUTC utc_time = 9;
DeviceIdentify device_identify = 10; // Include sn now.
}
// [end] Describe every data source as a message =============================
message DjiRealTimeTargetPoint {
int64 timestamp = 1;
uint32 frame_id = 2;
DjiUTC utc_time = 3;
repeated string source = 4; // The description of the one which create this point.
message DjiTargetPointStruct {
enum TARGET_POINT_TYPE {
NO_USED = 0;
TARGET_POINT = 1; // The target point create in local client.
RNG_POINT = 2; // The The laser ranging point is different from the target point.
// This point is the center point of the screen detected by laser ranging in real time.*/
ST_POINT = 3; // Spotlight tracking point, the target position calculated in the camera automatic tracking mode,
// on the screen display, mutually exclusive with Rng, if there is this point,
// the Rng point may not be displayed.
}
TARGET_POINT_TYPE type = 1; // The point type.
uint32 index = 2; // The point index.
double latitude = 3; // unit: rad.
double longitude = 4; // unit: rad.
float altitude_ellipsoid = 5; // unit: m. In ellipsoidal coordinate system.
float altitude_relative = 6; // unit: m. The altitude relatived to take-off point.
float video_stream_window_x = 7; // 0~1, Relative to the current position of the current stream viewport x,
// the upper left corner of the screen is 0, and negative numbers are used to indicate illegal values
// and are not displayed on the viewport.
float video_stream_window_y = 8; // 0~1, Relative to the current position of the current stream viewport x,
// the upper left corner of the screen is 0, and negative numbers are used to indicate illegal values
// and are not displayed on the viewport.
bool altitude_ellipsoid_can_use = 9;
}
repeated DjiTargetPointStruct TargetPointStruct = 5;
}
message DjiFlyingWindState {
int64 timestamp = 1;
uint32 frame_id = 2;
DjiUTC utc_time = 3;
float wind_speed = 4; // unit:m/s, precision:one decimal place
DIRECTION_TYPE wind_dir = 5; // The direction of the wind
}
message DjiFlyingHomePoint {
DjiGpsBasic point_pos_info = 3;
}
message DjiFlyingInherentInfo {
DjiInherentInfoBase base_info = 1;
}
message DjiFlyingFlyOrderID {
string order_id = 1; // limited to 48 characters
}
message DjiRCInherentInfo {
DjiInherentInfoBase base_info = 1;
enum REMOTE_CONTROL_ROLES {
RC_ROLES_A = 0;
RC_ROLES_B = 1;
}
REMOTE_CONTROL_ROLES rc_roles = 2; // [only remote control] The remote control position index
}
message DjiPayloadInherentInfo {
DIRECTION_TYPE base_info = 1;
enum PAYLOAD_PHY_POS {
INDEX_1 = 0;
INDEX_2 = 1;
INDEX_3 = 2;
}
PAYLOAD_PHY_POS payload_phy_pos = 2; // [only payload] The payload devices physical position index
}dji_video_metadata.proto
// DJI Media Metadata using Protobuf 3
// version = 1
syntax = "proto3";
import "dvtm_library.proto";
message DjiVideoMetadata {
DjiVideoGlobalInfo video_global_info = 1;
repeated DjiCameraBasic camera_basic = 5;
repeated DjiGpsBasic gps_basic = 6; // The old flying pos in camera, not suggest to use any more.
repeated DjiFlyingState flying_state = 7; // Push frequency: 10Hz, Include flying speed and flying attitude.
repeated DjiGimbal gimbal = 8; // Push frequency: Hz.
repeated DjiLaserRanging laser_ranging = 9;
DjiPayloadInherentInfo payload_inherent_info = 10; // Push frequency: 1Hz.
repeated DjiFlyingWindState flying_wind_state = 50; // Push frequency: 10Hz.
repeated DjiGpsBasic flying_pos = 51; // Push frequency: 10Hz, The position information of the flying, using GPS/RTK, etc.
repeated DjiFlyingHomePoint flying_home_point = 52; // Push frequency: 10Hz.
DjiRealTimeTargetPoint dji_rt_target_point = 53; // Push frequency: 2Hz.
DjiFlyingInherentInfo flying_inherent_info = 54; // Push frequency: 1Hz.
DjiFlyingFlyOrderID flying_order_id = 55; // Event trigger.
repeated DjiGpsBasic flying_rc_pos = 100; // Push frequency: Hz. The position information of the remote control which control this flying.
DjiRCInherentInfo flying_rc_inherent_info = 101; // Push frequency: 1Hz.
}Metadata 封装成一个单独 NALU，附加在每个 H264 视频 I 帧或 P 帧的末尾，NAL 类型为
0x06,其中 UUID 固定为
{0x81, 0x6d, 0x38, 0x4e, 0x99, 0x8c, 0x11, 0xea, 0xb2, 0x94, 0x02, 0xfc, 0xdc, 0x4e, 0x74, 0x12}3. Metadata 解析指导
- 将 metadata 数据，通过 protobuf 工具，序列化成二进制数据。
- 根据 Metadta in H264 介绍的格式，将 metadata protobuf 二进制数据封装到H264/H265 的 SEI 帧里，紧随着每一个 I/P 帧传输。
- 解析端收取的时候也是先收取到码流，根据码流解析出一个个帧，然后识别到 SEI帧，按照格式解析得到 metadata protobuf 二进制数据。最后根据 protobuf 格式(protobuf 支持多种语言多种平台)，解析出 metadata 数据结构体。
- 解析这个环节最关键的三个要素：metadata protobuf file，protobuf 对应平台的编译工具，H264/H265 SEI 帧。
四、DJI GB28181 协议信令详解
抽空再写
总结
本文大部分资源来自于大疆内部文档，本人仅对其中的文件缺失进行了补全，最终解释权归大疆所有
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
