swift ffmpeg 解码播放声音 ffmpeg解码mp3

转载

mob64ca14137e4f 2023-10-17 08:43:06

文章标签 swift ffmpeg 解码播放声音初始化数据创建线程 文章分类 Swift 移动开发

AudioTrackTest-ffmpeg

AudioTrackTest-ffmpeg

源码分析

大致的流程
1. NativeMp3Player

1. setDataSource进行初始化操作
2. prepare创建audiotrack和线程
3. start开始播放
4. stop停止播放

2. AccompanyDecoderController

1. init初始化ffmpeg，创建线程

1. macDecoder->init(acPath)
2. mpacketPool->initDecoderAccompanyPacketQueue
3. initDecoderThread()，启动accompany_decoder线程，进行解码，然后push audiopacket到队列中

1. macDecoder->decodePacket()
2. mpacketPool->pushDecoderAccompanyPacketQueue(acPacket)

2. readSamples，读线程，NativeMp3Player线程会来读取数据
1. mpacketPool->getDecoderAccompanyPacket(&acPacket, true)
3. destroy，停止的时候，进行清除工作

实用技巧

1. JNI中数组和字符串的使用方式
2. java中线程的使用方式
3. c++中线程的使用方式
4. lock和condition的使用方式
5. AudioTrack的使用流程
6. AudioTrack的内部实现流程（后面再结合看）

问题

1. audiotrack的play在write之前，可能会出错，造成无法播放的现象

参考

源码分析

大致的流程

该app一共有三个线程，一个主线程，一个NativeMp3Player线程，一个accompany_decoder线程。播放音乐的时候：

（1）NativeMp3Player线程从packet_pool中读audiopacket，然后将audiopacket中的buffer数据写到audiotrack中；

（2）而accompany_decoder线程先调用decodePacket对packet进行解码，得到pcm数据，然后打包成audiopacket，然后将audiopacket链接到packet_pool中。

1. NativeMp3Player

1. setDataSource进行初始化操作

public int setDataSource(String path) {
        mDecoder = new MusicDecoder();
        // accompany_decoder中的初始化操作，就是ffmpeg的初始化操作
        return mDecoder.init(path);
    }

2. prepare创建audiotrack和线程

public void prepare() {
        initPlayState();
        initAudioTrack();
        startPlayerThread();
    }
	private void initPlayState() {
        // 初始化播放的状态
        isPlaying = false;
        isStop = false;
    }

    private void startPlayerThread() {
        // 创建线程
        mPlayerThread = new Thread(new PlayerThread(), "NativeMp3PlayerThread");
        // 开始执行线程
        mPlayerThread.start();
    }

    private void initAudioTrack() {
        // 获得最小的buffer size
        int buffersize = AudioTrack.getMinBufferSize(44100, AudioFormat.CHANNEL_OUT_STEREO, AudioFormat.ENCODING_PCM_16BIT);
        // 创建audiotrack
        mAudioTrack = new AudioTrack.Builder()
                .setAudioAttributes(new AudioAttributes.Builder()
                        .setUsage(AudioAttributes.USAGE_MEDIA)
                        .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
                        .build())
                .setAudioFormat(new AudioFormat.Builder()
                        .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
                        .setSampleRate(44100)
                        .setChannelMask(AudioFormat.CHANNEL_OUT_STEREO)
                        .build())
                .setBufferSizeInBytes(buffersize)
                .build();
    }

    class PlayerThread implements Runnable {
        private short[] samples;

        @Override
        public void run() {
            int sample_count = 0;
            boolean isPlayTemp = isPlaying = false;
            try {
                // 按最大的192000来创建一个buffer，获得audiopacket数据
                samples = new short[DECODE_BUFFER_SIZE];
                int[] extraSlientSampleSize = new int[1];

                while (!isStop) {
                    extraSlientSampleSize[0] = 0;
                    // 读取pcm数据到buffer中
                    sample_count = mDecoder.readSamples(samples, extraSlientSampleSize);

                    if (sample_count == -2) {
                        try {
                            Thread.sleep(10);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                        continue;
                    }
                    if (sample_count < 0) {
                        break;
                    }

                    if (null != mAudioTrack && mAudioTrack.getState() != AudioTrack.STATE_UNINITIALIZED) {
                        // 写到audiotrack中
                        mAudioTrack.write(samples, 0, sample_count);
                    }

                    while (true) {
                        synchronized (NativeMp3Player.class) {
                            isPlayTemp = isPlaying;
                        }
                        if (isPlayTemp)
                            break;
                        else
                            Thread.yield();
                    }
                }
                mDecoder.destory();
            } catch (Error e) {
                e.printStackTrace();
            }
            samples = null;
        }
    }

3. start开始播放

public void start() {
        synchronized (NativeMp3Player.class) {
            try {
                if (null != mAudioTrack) {
                    // 开始播放
                    mAudioTrack.play();
                }
            } catch (Throwable t) {

            }
            isPlaying = true;
        }
    }

4. stop停止播放

public void stop() {
        if (!isStop && null != mAudioTrack) {
            if (null != mAudioTrack && mAudioTrack.getState() != AudioTrack.STATE_INITIALIZED) {
                try {
                    // stop停止播放
                    mAudioTrack.stop();
                } catch (Throwable t) {
                    t.printStackTrace();
                }
            }
            // 设置状态
            isPlaying = true;
            isStop = true;
            try {
                Log.i(TAG, "join decodeMp3Thread...");
                if (null != mPlayerThread) {
                    // 销毁线程
                    mPlayerThread.join();
                    mPlayerThread = null;
                }
                Log.e(TAG, "decodeMp3Thread ended....");
            } catch (Throwable t) {
                t.printStackTrace();
            }
            closeAudioTrack();
            destroy();
        }
    }

2. AccompanyDecoderController

1. init初始化ffmpeg，创建线程

int AccompanyDecoderController::init(const char *acPath) {
    LOGI("AccompanyDecoderController::Init");
    macDecoder = new AccompanyDecoder();
    // 1. 进行ffmpeg的初始化操作
    if (macDecoder->init(acPath) < 0) {
        LOGI("macDecoder->init fail...");
        return -1;
    } else {
        LOGI("macDecoder->init success...");
    }

    mpacketPool = PacketPool::GetInstance();
    // 2. 初始化audiopacket队列
    mpacketPool->initDecoderAccompanyPacketQueue();
    // 3. 初始化解码线程，该线程用来填充数据的
    initDecoderThread();
    return 0;
}

1. macDecoder->init(acPath)

int AccompanyDecoder::init(const char *fileString) {
    LOGI("enter AccompanyDecoder::init");
    mavFormatContext = NULL;
    mstream_index = -1;
    mavCodecContext = NULL;
    mswrContext = NULL;
    AVCodec * avCodec = NULL;
    mindex = 0;

    mpAudioFrame = av_frame_alloc();
    if (mpAudioFrame == NULL) {
        LOGI("av_frame_alloc mpAudioFrame fail...");
        return -1;
    } else {
        LOGI("av_frame_alloc mpAudioFrame success...");
    }
    maudioBuffer = (uint8_t *)av_malloc(MAX_AUDIO_FRAME_SIZE*2);
    if (maudioBuffer == NULL) {
        LOGI("av_malloc audiobuffer fail...");
        return -1;
    } else {
        LOGI("av_malloc audiobuffer success...");
    }
    // 注册解码器
    avcodec_register_all();
    av_register_all();
    mavFormatContext = avformat_alloc_context();
    LOGI("open ac file %s...", fileString);

    // 打开文件，并解析文件，然后填充avformat
    int result = avformat_open_input(&mavFormatContext, fileString, NULL, NULL);
    if (result != 0) {
        LOGI("can't open file %s result %s", fileString, av_err2str(result));
        return -1;
    } else {
        LOGI("open file %s success and result is %s", fileString, av_err2str(result));
    }

    // 检查文件中的流信息
    result = avformat_find_stream_info(mavFormatContext, NULL);
    if (result < 0) {
        LOGI("fail avformat avformat_find_stream_info %s result %s", fileString,
             av_err2str(result));
        return -1;
    } else {
        LOGI("avformat_find_stream_info success result is %s", fileString, av_err2str(result));
    }
    // 找stream
    mstream_index = av_find_best_stream(mavFormatContext, AVMEDIA_TYPE_AUDIO, -1, -1, NULL, 0);
    LOGI("stream_index is %d", mstream_index);
    if (mstream_index == -1) {
        LOGI("no audio stream");
        return -1;
    }
    mavCodecContext = mavFormatContext->streams[mstream_index]->codec;
    LOGI("avCodecContext->codec_id is %d AV_CODEC_ID_AAC is %d", mavCodecContext->codec_id,
         AV_CODEC_ID_AAC);
    // 找解码器
    avCodec = avcodec_find_decoder(mavCodecContext->codec_id);
    if (avCodec == NULL) {
        LOGI("Unsupported codec");
        return -1;
    }
    // 打开解码器
    result = avcodec_open2(mavCodecContext, avCodec, NULL);
    if (result < 0) {
        LOGI("avcodec_open2 fail avformat_find_stream_info result is %s", av_err2str(result));
        return -1;
    } else {
        LOGI("avcodec_open2 sucess avformat_find_stream_info result is %s", av_err2str(result));
    }
    if (!(mavCodecContext->sample_fmt == AV_SAMPLE_FMT_S16)) {
        LOGI("because of audio Codec Is Not Supported so we will init swresampler...");
        mswrContext = swr_alloc();
        // 采样率转换的设置
        mswrContext = swr_alloc_set_opts(mswrContext,
                                         av_get_default_channel_layout(OUT_PUT_CHANNELS),
                                         AV_SAMPLE_FMT_S16, 44100,
                                         av_get_default_channel_layout(mavCodecContext->channels),
                                         mavCodecContext->sample_fmt, mavCodecContext->sample_rate,
                                         0, NULL);
        if (!mswrContext || swr_init(mswrContext)) {
            if (mswrContext)
                swr_free(&mswrContext);
            avcodec_close(mavCodecContext);
            LOGI("init resampler failed...");
            return -1;
        }
    }
    LOGI(" channels is %d sampleRate is %d", mavCodecContext->channels,
         mavCodecContext->sample_rate);

    return 0;
}

2. mpacketPool->initDecoderAccompanyPacketQueue

void PacketPool::initDecoderAccompanyPacketQueue() {
    const char* name = "decoder accompany packet queue";
    // packetpool就是一个队列
    macPacketQueue = new PacketQueue(name);
}

PacketQueue::PacketQueue() {
    init();
}

void PacketQueue::init() {
    LOGI("enter PacketQueue init");
    // 队列需要锁来保护
    int initLockCode = pthread_mutex_init(&mLock, NULL);
    // 实现block机制，队列空了，就等待
    int initConditionCode = pthread_cond_init(&mCondition, NULL);
    // 队列元素的个数
    mNbPackets = 0;
    // 队列头指针
    mFirst = NULL;
    // 队列尾指针
    mLast = NULL;
    mAbortRequest = false;
}

3. initDecoderThread()，启动accompany_decoder线程，进行解码，然后push audiopacket到队列中

void AccompanyDecoderController::initDecoderThread() {
    LOGI("enter AccompanyDecoderController::initDecoderThread");
    // 设置状态
    isRunning = true;
    // 初始化锁和条件变量
    pthread_mutex_init(&mLock, NULL);
    pthread_cond_init(&mCondition, NULL);
    // 创建线程
    pthread_create(&songDecoderThread, NULL, startDecoderThread, this);
}

void* AccompanyDecoderController::startDecoderThread(void* ptr) {
    LOGI("enter AccompanyDecoderController::startDecoderThread");
    AccompanyDecoderController* decoderController = (AccompanyDecoderController *) ptr;
    int getLockCode = pthread_mutex_lock(&decoderController->mLock);
    while (decoderController->isRunning) {
        // 解码，获取包，链接到packet_pool中
        decoderController->decodeSongPacket();
        if (decoderController->mpacketPool->getDecoderAccompanyPacketQueueSize() > QUEUE_SIZE_MAX_THRESHOLD) {
            pthread_cond_wait(&decoderController->mCondition, &decoderController->mLock);
        }
    }
}

void AccompanyDecoderController::decodeSongPacket() {
    LOGI("AccompanyDecoderController::decodeSongPacket");
    // 1. 解码，获取audiopacket
    AudioPacket* acPacket = macDecoder->decodePacket();
    if (acPacket != NULL) {
        acPacket->maction = AudioPacket::AUDIO_PACKET_ACTION_PLAY;
        // 2. 链接到packet_pool中
        mpacketPool->pushDecoderAccompanyPacketQueue(acPacket);
    }
}

1. macDecoder->decodePacket()

AudioPacket* AccompanyDecoder::decodePacket() {
    int ret = 1;
    int gotframe = 0;
    AVSampleFormat outSampleFmt = AV_SAMPLE_FMT_S16;
    AudioPacket* samplePacket = NULL;

    av_init_packet(&mpacket);

    // 从文件中读取packet
    while (av_read_frame(mavFormatContext, &mpacket) >= 0) {
        //LOGI(" av_read_frame start");
        if (mpacket.stream_index == mstream_index) {
            //LOGI("av_read_frame mstream_index");
            // 从packet中解码到frame中
            if (avcodec_decode_audio4(mavCodecContext, mpAudioFrame, &gotframe, &mpacket) < 0) {
                LOGI("decode audio error, skip packet");
                return NULL;
            }
            //LOGI("av_read_frame avcodec_decode_audio4");
            if (gotframe) {
                //LOGI("av_read_frame gotframe");
                int outBufferSize=mpAudioFrame->nb_samples * OUT_PUT_CHANNELS;
                int numFrames = 0;
                // 重新采样
                if (mswrContext) {
                    // 进行重采样操作
                    numFrames = swr_convert(mswrContext, &maudioBuffer,
                                            mpAudioFrame->nb_samples,
                            (const uint8_t **)mpAudioFrame->data,
                            mpAudioFrame->nb_samples);
                    if (numFrames < 0) {
                        LOGI("fail resample audio");
                        return NULL;
                    }
                    LOGI("mindex:%5d\t pts:%lld\t packet size:%d out_buffer_size: %d\n",mindex,mpacket.pts,mpacket.size, outBufferSize);
                    // copy to AudioPacket
                    short * samples = new short[outBufferSize];
                    memcpy(samples, maudioBuffer, outBufferSize*2);
                    // 生成audiopacket，并返回
                    samplePacket = new AudioPacket();
                    if (samplePacket == NULL) {
                        return NULL;
                    }
                    samplePacket->mbuffer = samples;
                    samplePacket->msize = outBufferSize;
                    mindex++;
                    break;
                }
            }
        }
    }
    //LOGI(" end");
    av_packet_unref(&mpacket);

    return samplePacket;
}

2. mpacketPool->pushDecoderAccompanyPacketQueue(acPacket)

void PacketPool::pushDecoderAccompanyPacketQueue(AudioPacket *audioPacket) {
    macPacketQueue->put(audioPacket);
}

int PacketQueue::put(AudioPacket *audioPacket) {
    LOGI("enter PacketQueue put...");
    if (mAbortRequest) {
        delete audioPacket;
        return -1;
    }
    // 创建一个节点
    AudioPacketList *pkt1 = new AudioPacketList();
    if (!pkt1)
        return -1;
    pkt1->pkt = audioPacket;
    pkt1->next = NULL;

    // 对队列进行操作时，先获取锁
    int getLockCode = pthread_mutex_lock(&mLock);
    if (mLast == NULL) {
        // 如果队列为空
        mFirst = pkt1;
    } else {
        // 插入尾部
        mLast->next = pkt1;
    }

    mLast = pkt1;
    mNbPackets++;

    // 发送条件变量
    pthread_cond_signal(&mCondition);
    pthread_mutex_unlock(&mLock);
    return 0;
}

2. readSamples，读线程，NativeMp3Player线程会来读取数据

int AccompanyDecoderController::readSamples(short* samples, int size, int* slientSizeArr) {
    LOGI("AccompanyDecoderController::readSamples");
    int result = -1;
    AudioPacket* acPacket = NULL;
    // 1. 从队列中获取audiopacket包
    mpacketPool->getDecoderAccompanyPacket(&acPacket, true);
    if (NULL != acPacket) {
        int samplePacketSize = acPacket->msize;
        if (samplePacketSize != -1 && samplePacketSize <= size) {
            // 2. 将audiopacket中的数据拷贝到buffer中
            memcpy(samples, acPacket->mbuffer, samplePacketSize * 2);
            delete acPacket;
            result = samplePacketSize;
        }
    } else {
        result = -2;
    }
    // 3. 当队列中的包少于20个时，通知写线程，进行写操作
    if (mpacketPool->getDecoderAccompanyPacketQueueSize() < QUEUE_SIZE_MIN_THRESHOLD) {
        int getLockCode = pthread_mutex_lock(&mLock);
        if (result != -1) {
            pthread_cond_signal(&mCondition);
        }
        pthread_mutex_unlock(&mLock);
    }
    return result;
}

1. mpacketPool->getDecoderAccompanyPacket(&acPacket, true)

int PacketPool::getDecoderAccompanyPacket(AudioPacket **audioPacket, bool block) {
    int result = -1;
    if (NULL != macPacketQueue) {
        result = macPacketQueue->get(audioPacket, block);
    }
    return result;
}

int PacketQueue::get(AudioPacket **audioPacket, bool block) {
    LOGI("enter PacketQueue get...");
    AudioPacketList* pkt1;
    int ret;

    // 先获取锁
    int getLockCode = pthread_mutex_lock(&mLock);
    for (;;) {
        if (mAbortRequest) {
            ret = -1;
            break;
        }
        // 获取头部
        pkt1 = mFirst;
        if (pkt1) {
            // 指导下一个元素
            mFirst = pkt1->next;
            if (!mFirst)
                mLast = NULL;
            mNbPackets--;
            *audioPacket = pkt1->pkt;
            delete pkt1;
            pkt1 = NULL;
            ret = 1;
            break;
        } else if (!block) {
            ret = 0;
            break;
        } else {
            // 如果为空，则block着
            pthread_cond_wait(&mCondition, &mLock);
        }
    }

    pthread_mutex_unlock(&mLock);
    return ret;
}

3. destroy，停止的时候，进行清除工作

void AccompanyDecoderController::destroy() {
    LOGI("AccompanyDecoderController::Destroy");
    destroyDecoderThread();
    mpacketPool->abortDecoderAccompanyPacketQueue();
    mpacketPool->destoryDecoderAccompanyPacketQueue();
    if (NULL != macDecoder) {
        macDecoder->destroy();
        delete macDecoder;
        macDecoder = NULL;
    }
}

实用技巧

1. JNI中数组和字符串的使用方式

jshort* target = env->GetShortArrayElements(array, 0);
        jint* slientSizeArr = env->GetIntArrayElements(extraSlientSampleSize, 0);
        int result = mDecoderController->readSamples(target, size, slientSizeArr);
        env->ReleaseIntArrayElements(extraSlientSampleSize, slientSizeArr, 0);
        env->ReleaseShortArrayElements(array, target, 0);
        
        const char * acFilePath = env->GetStringUTFChars(acFilePathParam, NULL);
        mDecoderController->init(acFilePath);
        env->ReleaseStringUTFChars(acFilePathParam, acFilePath);

2. java中线程的使用方式

// 创建线程，线程名
		mPlayerThread = new Thread(new PlayerThread(), "NativeMp3PlayerThread");
		// 然后调用start函数
        mPlayerThread.start();

// 继承Runnable
class PlayerThread implements Runnable {
        @Override
        public void run() {}
}

3. c++中线程的使用方式

// 创建pthread_t变量
pthread_t songDecoderThread;
// 执行startDecoderThread函数，传递this参数
pthread_create(&songDecoderThread, NULL, startDecoderThread, this);
// 销毁线程
pthread_join(songDecoderThread, &status);

4. lock和condition的使用方式

// 首先定义变量
	pthread_mutex_t mLock;
    pthread_cond_t mCondition;
	// 然后进行初始化
    pthread_mutex_init(&mLock, NULL);
    pthread_cond_init(&mCondition, NULL);
	// 使用，lock和unlock
	pthread_mutex_lock(&mLock);
	pthread_cond_signal(&mCondition);
    pthread_mutex_unlock(&mLock);
	// 最后进行销毁操作
	pthread_mutex_destroy(&mLock);
    pthread_cond_destroy(&mCondition);

5. AudioTrack的使用流程

// 先获取最小buffer
// 然后创建audiotrack
int buffersize = AudioTrack.getMinBufferSize(44100, AudioFormat.CHANNEL_OUT_STEREO, AudioFormat.ENCODING_PCM_16BIT);
        mAudioTrack = new AudioTrack.Builder()
                .setAudioAttributes(new AudioAttributes.Builder()
                        .setUsage(AudioAttributes.USAGE_MEDIA)
                        .setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
                        .build())
                .setAudioFormat(new AudioFormat.Builder()
                        .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
                        .setSampleRate(44100)
                        .setChannelMask(AudioFormat.CHANNEL_OUT_STEREO)
                        .build())
                .setBufferSizeInBytes(buffersize)
                .build();
// write是一个循环操作
// 写数据
mAudioTrack.write(samples, 0, sample_count);
// 播放只需要调用一次
mAudioTrack.play();

mAudioTrack.stop();
mAudioTrack.release();

6. AudioTrack的内部实现流程（后面再结合看）

问题

1. audiotrack的play在write之前，可能会出错，造成无法播放的现象

write和play之间存在一个时差问题，把write放在onCreate的时候，然后按下按钮的时候，就play。

参考

1. 源码位置：https://github.com/mashenlyl/AudioTrackTest

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。

上一篇：芯烨 java sdk 芯烨打印机蓝牙

下一篇：android11下载图片到相册 android 下载图片

提问和评论都可以，用心的回复会被更多人看到评论

发布评论

相关文章

官方博客	全部文章	热门标签	班级博客
了解我们	网站地图	意见反馈

鸿蒙开发者社区	51CTO学堂
51CTO	软考资讯

swift ffmpeg 解码播放声音 ffmpeg解码mp3

swift ffmpeg 解码播放声音 ffmpeg解码mp3

AudioTrackTest-ffmpeg

源码分析

大致的流程

1. NativeMp3Player

1. setDataSource进行初始化操作

2. prepare创建audiotrack和线程

3. start开始播放

4. stop停止播放

2. AccompanyDecoderController

1. init初始化ffmpeg，创建线程

1. macDecoder->init(acPath)

2. mpacketPool->initDecoderAccompanyPacketQueue

3. initDecoderThread()，启动accompany_decoder线程，进行解码，然后push audiopacket到队列中

1. macDecoder->decodePacket()

2. mpacketPool->pushDecoderAccompanyPacketQueue(acPacket)

2. readSamples，读线程，NativeMp3Player线程会来读取数据

1. mpacketPool->getDecoderAccompanyPacket(&acPacket, true)

3. destroy，停止的时候，进行清除工作

实用技巧

1. JNI中数组和字符串的使用方式

2. java中线程的使用方式

3. c++中线程的使用方式

4. lock和condition的使用方式

5. AudioTrack的使用流程

6. AudioTrack的内部实现流程（后面再结合看）

问题

1. audiotrack的play在write之前，可能会出错，造成无法播放的现象

参考

51CTO博客