Android Camera HAL3 分析资料报告

wordAndroid Camera HAL3 分析本文均属自己阅读源码的点滴总结，转账请注明出处谢谢。欢迎和大家交流。 :1037701636 email:gzzaig2009163 Software：系统源码Android5.1Camera3研读前沿：当初在研读Camera1.0相关的内容时，主要围绕着CameraClient、CameraHardwareInterface等方面进展工作的开展，无论是数据流还是控制流看起来都很简单、明了，一系列的流程化操作使得整个框架学起来特别的容易。因为没有Camera2.0相关的根底，所以这次直接看3.0相关的源码时，显得十分的吃紧，再加上底层高通HAL3.0实现的过程也是相当的复杂，都给整个研读过程带来了很多的困难。可以说，自身目前对Camera3.0框架的熟悉度也大概只有70%左右，希望通过总结来进一步梳理他的工作原理与整个框架，并进一步熟悉与加深理解。1.Camera3下的整体架构图。整个CameraService建立起一个可用操作底层Camera device大致需要经过Camera2Client、Camera3Device以与HAL层的camera3_device_t三个局部。从上图中可以发现Camera3架构看上去明显比camera1来的复杂，但他更加的模块化。比照起Android4.2.2 Camer系统架构图HAL和回调处理一文中描述的单顺序执行流程，Camera3将更多的工作集中在了Framework去完成，将更多的控制权掌握在自己的手里，从而与HAL的交互的数据信息更少，也进一步减轻了一些在旧版本中HAL层所需要做的事情。2. Camera2Client的建立与初始化过程在建立好Camera2Client后会进展initialize操作，完成各个处理模块的创建：?12345678910111213141516171819202122232425.mStreamingProcessor = newStreamingProcessor(this);/preview和recorderthreadName = String8:format(C2-%d-StreamProc,mCameraId);mStreamingProcessor-run(threadName.string();/预览与录像mFrameProcessor = newFrameProcessor(mDevice, this);/ 3AthreadName = String8:format(C2-%d-FrameProc,mCameraId);mFrameProcessor-run(threadName.string(); /3AmCaptureSequencer = newCaptureSequencer(this);threadName = String8:format(C2-%d-CaptureSeq,mCameraId);mCaptureSequencer-run(threadName.string();/录像，拍照mJpegProcessor = newJpegProcessor(this, mCaptureSequencer);threadName = String8:format(C2-%d-JpegProc,mCameraId);mJpegProcessor-run(threadName.string();.mCallbackProcessor = newCallbackProcessor(this);/回调处理threadName = String8:format(C2-%d-CallbkProc,mCameraId);mCallbackProcessor-run(threadName.string();依次分别创建了：StreamingProcessor并启动一个他所属的thread，该模块主要负责处理previews与record两种视频流的处理，用于从hal层获取原始的视频数据FrameProcessor并启动一个thread，该模块专门用于处理回调回来的每一帧的3A等信息，即每一帧视频除去原始视频数据外，还应该有其他附加的数据信息，如3A值。CaptureSequencer并启动一个thread，该模块需要和其他模块配合使用，主要用于向APP层告知capture到的picture。 JpegProcessor并启动一个thread，该模块和streamprocessor类似，他启动一个拍照流，一般用于从HAL层获取jpeg编码后的图像照片数据。此外ZslProcessor模块称之为0秒快拍，其本质是直接从原始的Preview流中获取预存着的最近的几帧，直接编码后返回给APP，而不需要再经过take picture去请求获取jpeg数据。0秒快拍技术得意于当下处理器CSI2 MIPI性能的提升以与Sensor支持全像素高帧率的实时输出。一般手机拍照在按下快门后都会有一定的延时，是因为需要切换底层Camera以与ISP等的工作模式，并重新设置参数以与重新对焦等等，都需要花一定时间后才抓取一帧用于编码为jpeg图像。以上5个模块整合在一起根本上实现了Camera应用开发所需的根本业务功能。3. 预览Preview下的控制流研读Camera具体的业务处理功能，一般从视频实时预览Preview入手。一般熟悉Camera架构的人，可以从一个app端的一个api一直连续打通到底层hal的一个控制命令。大致可以如如下图所示：对于preview局部到CameraService的控制流可以参考博文Android4.2.2的preview的数据流和控制流以与最终的预览显示，本文将直接从Camera2Client:startPreview() 作为入口来分析整个Framework层中Preview相关的数据流。?123456789status_t Camera2Client:startPreview() ATRACE_CALL();ALOGV(%s: E, _FUNCTION_);Mutex:Autolock icl(mBinderSerializationLock);status_t res;if( (res = checkPid(_FUNCTION_) ) != OK) returnres;SharedParameters:Lock l(mParameters);returnstartPreviewL(l.mParameters, false);startPreview通过startPreviewL提取参数后真正的开始执行Preview相关的控制流。该函数看上去内容虽然较多，但根本采用了同一种处理方式：?123456789101112131415161718192021222324252627282930313233343536373839404142434445464748status_t Camera2Client:startPreviewL(Parameters ms, bool restart) /restart = falseATRACE_CALL();status_t res;.int lastPreviewStreamId = mStreamingProcessor-getPreviewStreamId();/获取上一层Preview stream idres = mStreamingProcessor-updatePreviewStream(params);/创建camera3device stream， Camera3OutputStream.int lastJpegStreamId = mJpegProcessor-getStreamId();res = updateProcessorStream(mJpegProcessor, params);/预览启动时就建立一个jpeg的outstream.res = mCallbackProcessor-updateStream(params);/回调处理建立一个Camera3outputstreamif (res != OK) ALOGE(%s: Camera %d: Unable to update callback stream: %s (%d),_FUNCTION_, mCameraId, strerror(-res), res);return res;outputStreams.push(getCallbackStreamId();.outputStreams.push(getPreviewStreamId();/预览stream.if (!params.recordingHint) if (!restart) res = mStreamingProcessor-updatePreviewRequest(params);/request处理,更新了mPreviewrequestif (res != OK) ALOGE(%s: Camera %d: Cant set up preview request: %s (%d), _FUNCTION_, mCameraId,strerror(-res), res);return res;res = mStreamingProcessor-startStream(StreamingProcessor:PREVIEW,outputStreams);/启动stream，传入outputStreams即stream 的id else if (!restart) res = mStreamingProcessor-updateRecordingRequest(params);if (res != OK) ALOGE(%s: Camera %d: Cant set up preview request with record hint: %s (%d), _FUNCTION_, mCameraId,strerror(-res), res);return res;res = mStreamingProcessor-startStream(StreamingProcessor:RECORD,outputStreams);.(1). mStreamingProcessor-updatePreviewStream()由预览与录像处理模块更新一个预览流，其实现过程如下：?1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465status_t StreamingProcessor:updatePreviewStream(constParameters ms) ATRACE_CALL();Mutex:Autolock m(mMutex);status_t res;sp device = mDevice.promote();/Camera3Deviceif(device = 0) ALOGE(%s: Camera %d: Device does not exist, _FUNCTION_, mId);returnINVALID_OPERATION;if(mPreviewStreamId != NO_STREAM) / Check if stream parameters have to changeuint32_t currentWidth, currentHeight;res = device-getStreamInfo(mPreviewStreamId,tWidth, tHeight, 0);if(res != OK) ALOGE(%s: Camera %d: Error querying preview stream info: %s (%d), _FUNCTION_, mId, strerror(-res), res);returnres;if(currentWidth != (uint32_t)params.previewWidth |currentHeight != (uint32_t)params.previewHeight) ALOGV(%s: Camera %d: Preview size switch: %d x %d - %d x %d,_FUNCTION_, mId, currentWidth, currentHeight,params.previewWidth, params.previewHeight);res = device-waitUntilDrained();if(res != OK) ALOGE(%s: Camera %d: Error waiting forpreview to drain: %s (%d), _FUNCTION_, mId, strerror(-res), res);returnres;res = device-deleteStream(mPreviewStreamId);if(res != OK) ALOGE(%s: Camera %d: Unable to delete old output stream forpreview: %s (%d), _FUNCTION_, mId,strerror(-res), res);returnres;mPreviewStreamId = NO_STREAM;if(mPreviewStreamId = NO_STREAM) /首次create streamres = device-createStream(mPreviewWindow,params.previewWidth, params.previewHeight,CAMERA2_HAL_PIXEL_FORMAT_OPAQUE, &mPreviewStreamId);/创建一个Camera3OutputStreamif(res != OK) ALOGE(%s: Camera %d: Unable to create preview stream: %s (%d),_FUNCTION_, mId, strerror(-res), res);returnres;res = device-setStreamTransform(mPreviewStreamId,params.previewTransform);if(res != OK) ALOGE(%s: Camera %d: Unable to set preview stream transform: %s (%d), _FUNCTION_, mId, strerror(-res), res);returnres;returnOK;该函数首先是查看当前StreamingProcessor模块下是否存在Stream，没有的话，如此交由Camera3Device创建一个stream。显然，一个StreamingProcessor只能拥有一个PreviewStream，而一个Camera3Device显然控制着所有的Stream。注意：在Camera2Client中，Stream大行其道，5大模块的数据交互均以stream作为根底。下面我们来重点关注Camera3Device的接口createStream，他是5个模块创建stream的根底：?12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576status_t Camera3Device:createStream(sp consumer,uint32_t width, uint32_t height, intformat, int*id) ATRACE_CALL();Mutex:Autolock il(mInterfaceLock);Mutex:Autolock l(mLock);ALOGV(Camera %d: Creating newstream %d: %d x %d, format %d,mId, mNextStreamId, width, height, format);status_t res;bool wasActive = false;switch(mStatus) caseSTATUS_ERROR:CLOGE(Device has encountered a serious error);returnINVALID_OPERATION;caseSTATUS_UNINITIALIZED:CLOGE(Device not initialized);returnINVALID_OPERATION;caseSTATUS_UNCONFIGURED:caseSTATUS_CONFIGURED:/ OKbreak;caseSTATUS_ACTIVE:ALOGV(%s: Stopping activity to reconfigure streams, _FUNCTION_);res = internalPauseAndWaitLocked();if(res != OK) SET_ERR_L(Cant pause captures to reconfigure streams!);returnres;wasActive = true;break;default:SET_ERR_L(Unexpected status: %d, mStatus);returnINVALID_OPERATION;assert(mStatus != STATUS_ACTIVE);sp newStream;if(format = HAL_PIXEL_FORMAT_BLOB) /图片ssize_t jpegBufferSize = getJpegBufferSize(width, height);if(jpegBufferSize setStatusTracker(mStatusTracker);res = mOutputStreams.add(mNextStreamId, newStream);/一个streamid与Camera3OutputStream绑定if(res 0) SET_ERR_L(Cant add newstream to set: %s (%d), strerror(-res), res);returnres;*id = mNextStreamId+;/至少一个previewstream 一般还有CallbackStreammNeedConfig = true;/ Continue captures if active at startif(wasActive) ALOGV(%s: Restarting activity to reconfigure streams, _FUNCTION_);res = configureStreamsLocked();if(res != OK) CLOGE(Cant reconfigure device fornewstream %d: %s (%d),mNextStreamId, strerror(-res), res);returnres;internalResumeLocked();ALOGV(Camera %d: Created newstream, mId);returnOK;该函数重点是关注一个new Camera3OutputStream，在Camera3Device主要存在Camera3OutputStream和Camera3InputStream两种stream，前者主要作为HAL的输出，是请求HAL填充数据的OutPutStream，后者是由Framework将Stream进展填充。无论是Preview、record还是capture均是从HAL层获取数据，故都会以OutPutStream的形式存在，是我们关注的重点，后面在描述Preview的数据流时还会进一步的阐述。每当创建一个OutPutStream后，相关的stream信息被push维护在一个mOutputStreams的KeyedVector表中，分别是该stream在Camera3Device中创建时的ID以与Camera3OutputStream的sp值。同时对mNextStreamId记录下一个Stream的ID号。上述过程完成StreamingProcessor模块中一个PreviewStream的创建，其中Camera3OutputStream创建时的ID值被返回记录作为mPreviewStreamId的值，此外每个Stream都会有一个对应的ANativeWindow，这里称之为Consumer。2mCallbackProcessor-updateStream(params)比照StreamingProcessor模块创建previewstream的过程，很容易定位到Callback模块是需要建立一个callback流，同样需要创建一个Camera3OutputStream来接收HAL返回的每一帧帧数据，是否需要callback可以通过callbackenable来控制。一般但预览阶段可能不需要回调每一帧的数据到APP，但涉与到相应的其他业务如视频处理时，就需要进展callback的enable。?1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283status_t CallbackProcessor:updateStream(constParameters ms) ATRACE_CALL();status_t res;Mutex:Autolock l(mInputMutex);sp device = mDevice.promote();if(device = 0) ALOGE(%s: Camera %d: Device does not exist, _FUNCTION_, mId);returnINVALID_OPERATION;/ If possible, use the flexible YUV formatint32_t callbackFormat = params.previewFormat;if(mCallbackToApp) / TODO: etalvala: This should use the flexible YUV format as well, but/ need to reconcile HAL2/HAL3 requirements.callbackFormat = HAL_PIXEL_FORMAT_YV12; elseif(params.fastInfo.useFlexibleYuv &(params.previewFormat = HAL_PIXEL_FORMAT_YCrCb_420_SP |params.previewFormat = HAL_PIXEL_FORMAT_YV12) ) callbackFormat = HAL_PIXEL_FORMAT_YCbCr_420_888;if(!mCallbackToApp & mCallbackConsumer = 0) / Create CPU buffer queue endpoint, since app hasnt given us one/ Make it async to avoid disconnect deadlockssp producer;sp consumer;BufferQueue:createBufferQueue(&producer, &consumer);/BufferQueueProducer与BufferQueueConsumermCallbackConsumer = newCpuConsumer(consumer, kCallbackHeapCount);mCallbackConsumer-setFrameAvailableListener(this);/当前CallbackProcessor继承于CpuConsumer:FrameAvailableListenermCallbackConsumer-setName(String8(Camera2Client:CallbackConsumer);mCallbackWindow = newSurface(producer);/用于queue操作，这里直接进展本地的buffer操作if(mCallbackStreamId != NO_STREAM) / Check if stream parameters have to changeuint32_t currentWidth, currentHeight, currentFormat;res = device-getStreamInfo(mCallbackStreamId,tWidth, tHeight, tFormat);if(res != OK) ALOGE(%s: Camera %d: Error querying callback output stream info: %s (%d), _FUNCTION_, mId,strerror(-res), res);returnres;if(currentWidth != (uint32_t)params.previewWidth |currentHeight != (uint32_t)params.previewHeight |currentFormat != (uint32_t)callbackFormat) / Since size should only change while preview is not running,/ assuming that all existing use of old callback stream is/ pleted.ALOGV(%s: Camera %d: Deleting stream %d since the buffer parameters changed, _FUNCTION_, mId, mCallbackStreamId);res = device-deleteStream(mCallbackStreamId);if(res != OK) ALOGE(%s: Camera %d: Unable to delete old output stream forcallbacks: %s (%d), _FUNCTION_,mId, strerror(-res), res);returnres;mCallbackStreamId = NO_STREAM;if(mCallbackStreamId = NO_STREAM) ALOGV(Creating callback stream: %d x %d, format 0x%x, API format 0x%x,params.previewWidth, params.previewHeight,callbackFormat, params.previewFormat);res = device-createStream(mCallbackWindow,params.previewWidth, params.previewHeight,callbackFormat, &mCallbackStreamId);/Creating callback streamif(res != OK) ALOGE(%s: Camera %d: Cant create output stream forcallbacks: %s (%d), _FUNCTION_, mId,strerror(-res), res);returnres;returnOK;比照updatePreviewStream可以发现，该函数自助创建了一套surface/BufferQueue/CpuConsumer的机制，这套类似SurfaceFlinger的buffer管理机制可参看一文Android5.1中surface生产者和消费者间的处理框架简述。此外通过createStream请求Camera3Device建立一个Stream，其中Stream的ID值保存在mCallBackStreamId当中，并将一个CallbackWindow和当前的Stream绑定。通过这个比照，也需要重点关注到，对于每个Camera3OutPutStream来说，每一个stream都被一个Consumer，而在此处都是Surface(ANativeWindow)所拥有，这个Consumer和HAL相匹配来说是消费者，但对于真正的处理Buffer的Consumer来说如CPUConsumer，Surface却又是以一个Product的角色存在的。3updateProcessorStream(mJpegProcessor, params)?12345678910111213141516status_t Camera2Client:updateProcessorStream(sp processor,camera2:Parameters params) / No default template arguments until C+11, so we need this overloadreturnupdateProcessorStream(processor, params);template status_t Camera2Client:updateProcessorStream(sp processor,Parameters params) status_t res;/ Get raw pointer since sp doesnt have operator-*ProcessorT *processorPtr = processor.get();res = (processorPtr-*updateStreamF)(params);.该模板函数处理过程最终通过非显示实例到显示实例调用JpegProcessor:updateStream，该函数处理的逻辑根本和Callback模块处理一致，创建的一个OutPutStream和CaptureWindow相互绑定，同时Stream的ID保存在mCaptureStreamId中。此外需要说明一点：在preview模式下，就去创建一个jpeg处理的stream，目的在于启动takepicture时，可以更快的进展capture操作。是通过牺牲内存空间来提升效率。4整合startPreviewL中所有的stream 到VectoroutputStreamsoutputStreams.push(getPreviewStreamId();/预览streamoutputStreams.push(getCallbackStreamId()/Callback stream目前一次Preview构建的stream数目至少为两个。5mStreamingProcessor-updatePreviewRequest()在创建好多路stream后，由StreamingProcessor模块来将所有的stream信息交由Camera3Device去打包成Request请求。注意：Camera HAL2/3的特点是：将所有stream的请求都转化为几个典型的Request请求，而这些Request需要由HAL去解析，进而处理所需的业务。这也是Camera3数据处理复杂化的原因所在。?123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657status_t StreamingProcessor:updatePreviewRequest(constParameters ms) ATRACE_CALL();status_t res;sp device = mDevice.promote();if(device = 0) ALOGE(%s: Camera %d: Device does not exist, _FUNCTION_, mId);returnINVALID_OPERATION;Mutex:Autolock m(mMutex);if(mPreviewRequest.entryCount() = 0) sp client = mClient.promote();if(client = 0) ALOGE(%s: Camera %d: Client does not exist, _FUNCTION_, mId);returnINVALID_OPERATION;/ Use CAMERA3_TEMPLATE_ZERO_SHUTTER_LAG for ZSL streaming case.if(client-getCameraDeviceVersion() = CAMERA_DEVICE_API_VERSION_3_0) if(params.zslMode & !params.recordingHint) res = device-createDefaultRequest(CAMERA3_TEMPLATE_ZERO_SHUTTER_LAG,&mPreviewRequest); elseres = device-createDefaultRequest(CAMERA3_TEMPLATE_PREVIEW,&mPreviewRequest); elseres = device-createDef