Чтение из файла .oni приводит к неправильным кадрам

Моя проблема является дубликатом этот. Там не было никакого решения. В надежде найти решение и подробно описать мои конкретные настройки, ниже показана функция, используемая для чтения кадров из файла .oni. Если эта функция запускается с Type == 2 (то есть выполняется для # RGBD-изображений, где Criteria — это #), запуск этой функции в цикле for должен позволить пользователю получить доступ к каждому изображению. Однако индексы для цветного изображения и изображения глубины не совпадают и не соответствуют порядку. Это продолжается до тех пор, пока waitForAnyStream не истечет время для всех следующих вызовов IMG_pipeline :: listen (…).

void IMG_pipeline::listen(int Type, int Criteria){
int exitNumber;
clock_t start = clock();
double elapsedtime;
openni::VideoFrameRef frame;
int CurrentIMGCount=0;

switch (Type){
case 0:
{
exitNumber = -1;
break;
}
case 1:
{
exitNumber = Criteria;
break;
}
case 2:
{
exitNumber = -1;
break;
}
}for (int i = 0;i!=exitNumber;i++){
readyStream = -1;
rc = openni::OpenNI::waitForAnyStream(streams, 2, &readyStream, SAMPLE_READ_WAIT_TIMEOUT);
if (rc != openni::STATUS_OK)
{
printf("Wait failed! (timeout is %d ms)\n%s\n", SAMPLE_READ_WAIT_TIMEOUT, openni::OpenNI::getExtendedError());
//break;
}

switch (readyStream)
{
case 0:
{
// Depth
depth.readFrame(&frame);
break;
}
case 1:
{
// Color
color.readFrame(&frame);
break;
}
default:
{
printf("Unexpected stream: %i\n", readyStream);
continue;
}
}int Height = frame.getHeight();
int Width = frame.getWidth();

cvColor.release();
cvX.release();
cvY.release();
cvZ.release();

cvColor = cv::Mat(Height, Width, CV_8UC3);
cvX = cv::Mat(Height, Width, CV_32F);
cvY = cv::Mat(Height, Width, CV_32F);
cvZ = cv::Mat(Height, Width, CV_32F);

switch (frame.getVideoMode().getPixelFormat())
{
case openni::PIXEL_FORMAT_DEPTH_1_MM:
case openni::PIXEL_FORMAT_DEPTH_100_UM:
{
openni::DepthPixel* pDepth = (openni::DepthPixel*)frame.getData();
int k =0;
for (int ri = 0; ri<Height; ri++)
{
for (int ci = 0; ci<Width; ci++)
{
float pdepth_val = pDepth[k];
openni::CoordinateConverter::convertDepthToWorld(depth, (float)ri, (float)ci, pdepth_val, &cvX.at<float>(ri,ci), &cvY.at<float>(ri,ci), &cvZ.at<float>(ri,ci));
k++;
}
}
TotalFrames[0]++;
XYZCaptured = true;
printf("Frame Index: %i \n", frame.getFrameIndex());
printf("Depth Captured. \n");
break;
}
case openni::PIXEL_FORMAT_RGB888:
{
cvColor.data = (uchar*)frame.getData();
TotalFrames[1]++;
ColorCaptured = true;
printf("Frame Index: %i \n", frame.getFrameIndex());
printf("Color Captured. \n");
break;
}
default:
printf("Unknown format \n");
}

printf("Frame extracted. \n");
if (ColorCaptured && XYZCaptured){
if (NewButNotRead == true){
IMGsMissed++;}
else
NewButNotRead = true;

ColorCaptured = false;
XYZCaptured = false;
RGBD_out.clear();
RGBD_out.push_back(cvX);
RGBD_out.push_back(cvY);
RGBD_out.push_back(cvZ);
RGBD_out.push_back(cvColor);
CurrentIMGCount++;
printf("Image overwritten. \n");

}
elapsedtime=(clock()-start)/((double)CLOCKS_PER_SEC);

printf("Time since listen initiation: %f \n \n", elapsedtime);
if (CurrentIMGCount ==Criteria && Type == 2)
return;
else if (elapsedtime>(double)Criteria && Type==0)
return;

}
frame.release();

}

Вот пример вывода консоли:

Frame Index: 1
Depth Captured.
Frame extracted.
Time since listen initiation: 0.004846

Frame Index: 2
Depth Captured.
Frame extracted.
Time since listen initiation: 0.011601

Frame Index: 1
Color Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.012640

Depth frame count: 3
Color frame count: 2
Frame Index: 54
Color Captured.
Frame extracted.
Time since listen initiation: 0.000067

Frame Index: 57
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.005878

Depth frame count: 4
Color frame count: 3
Frame Index: 96
Color Captured.
Frame extracted.
Time since listen initiation: 0.000079

Frame Index: 99
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.003628

Depth frame count: 5
Color frame count: 4
Frame Index: 126
Color Captured.
Frame extracted.
Time since listen initiation: 0.000048

Frame Index: 130
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004782

Depth frame count: 6
Color frame count: 5
Frame Index: 152
Color Captured.
Frame extracted.
Time since listen initiation: 0.000065

Frame Index: 156
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.008294

Depth frame count: 7
Color frame count: 6
Frame Index: 181
Color Captured.
Frame extracted.
Time since listen initiation: 0.000045

Frame Index: 185
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004095

Depth frame count: 8
Color frame count: 7
Frame Index: 208
Color Captured.
Frame extracted.
Time since listen initiation: 0.000054

Frame Index: 212
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004242

Depth frame count: 9
Color frame count: 8
Frame Index: 236
Color Captured.
Frame extracted.
Time since listen initiation: 0.000092

Frame Index: 240
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.005918

Depth frame count: 10
Color frame count: 9
Frame Index: 261
Color Captured.
Frame extracted.
Time since listen initiation: 0.000731

Frame Index: 262
Color Captured.
Frame extracted.
Time since listen initiation: 0.000877

Frame Index: 266
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.009347

Depth frame count: 11
Color frame count: 11
Frame Index: 286
Color Captured.
Frame extracted.
Time since listen initiation: 0.000047

Frame Index: 290
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.006080

Depth frame count: 12
Color frame count: 12
Frame Index: 311
Color Captured.
Frame extracted.
Time since listen initiation: 0.000072

Frame Index: 315
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.006453

Depth frame count: 13
Color frame count: 13
Frame Index: 337
Color Captured.
Frame extracted.
Time since listen initiation: 0.000062

Frame Index: 341
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007485

Depth frame count: 14
Color frame count: 14
Frame Index: 367
Color Captured.
Frame extracted.
Time since listen initiation: 0.000042

Frame Index: 371
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.003758

Depth frame count: 15
Color frame count: 15
Frame Index: 390
Color Captured.
Frame extracted.
Time since listen initiation: 0.000073

Frame Index: 395
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007917

Depth frame count: 16
Color frame count: 16
Frame Index: 416
Color Captured.
Frame extracted.
Time since listen initiation: 0.000105

Frame Index: 421
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007554

Depth frame count: 17
Color frame count: 17
Frame Index: 453
Color Captured.
Frame extracted.
Time since listen initiation: 0.000060

Frame Index: 458
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.006150

Depth frame count: 18
Color frame count: 18
Frame Index: 481
Color Captured.
Frame extracted.
Time since listen initiation: 0.000074

Frame Index: 486
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007169

Depth frame count: 19
Color frame count: 19
Frame Index: 517
Color Captured.
Frame extracted.
Time since listen initiation: 0.000045

Frame Index: 522
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004196

Depth frame count: 20
Color frame count: 20
Frame Index: 547
Color Captured.
Frame extracted.
Time since listen initiation: 0.000071

Frame Index: 552
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007375

Depth frame count: 21
Color frame count: 21
Frame Index: 625
Color Captured.
Frame extracted.
Time since listen initiation: 0.000179

Frame Index: 631
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007922

Depth frame count: 22
Color frame count: 22
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached

Unexpected stream: -1

А вот и вызов IMG_pipeline :: listen (…):

IMG_pipeline pip_inst;
std::string FileName = "/home/derek/Test Data/RGBD/RGBD_S2_R1";
int Type = 2;
int Criteria = 1;
std::vector<cv::Mat> OUT;
int NumMissedIMGs;

int Start;
int Stop;

pip_inst.connect(FileName);

while (true)
{
pip_inst.listen(Type, Criteria);
if (pip_inst.IsNewIMG()){
OUT = pip_inst.GetImage();
cv::imshow("Current Frame", OUT.at(3));
char c = cv::waitKey(0);
if (c == 'f')
{
printf("Depth frame count: %i \n", pip_inst.GetDepthFrameCount());
printf("Color frame count: %i \n", pip_inst.GetColorFrameCount());

}
else
{
Start = pip_inst.GetColorFrameCount();
break;
}
cv::destroyWindow("Current Frame");

}}

Цветные изображения также чередуются оттенки R, G, B. Я уверен, что проблема в том, в каком порядке находятся данные в cv :: Mat.

Еще интереснее то, что вызов IMG_pipeline :: listen (…), который проходит через много кадров, имеет другой индексный результат, чем при запуске IMG_pipeline :: listen (…) несколько раз, с шагом в файл .oni.