Download

  1. Home
  2. Download

Download

To request the dataset, please sign the downloaded License Agreement here, and drop an e-mail (with your institution's mail address) to Huiwen Luo (huiwenluo05@gmail.com) and CC Wayne Wu(wuwenyan0503@gmail.com) and Kwan-Yee Lin(linjunyi9335@gmail.com).
*Do not give one word answers such as 'testing' or 'research'. Please give a more detailed description (2~3sentences) of the intended usage. It would help us to process your request more efficiently. We'll reply the requests on every Wednesday and Friday. Thanks for your patience.

Please read the ReadMe.txt in the shared link first, which provides a brief introduction to each script's function and every folder's structure.

We recommend users to check preview videos before downloading the whole motion sequences, because whole data takes up a lot of storage space.

If you have any other questions, please discuss here: FAQ

Overview

DNA-Rendering dataset contains 500 actors, 1500 outfits and 1187 motion types,and we captured over 5000 motion sequences in total.
In our technical report , part 1 (39 cases) is used for case specific benchmarks in Novel View Synthesis and Novel Pose Animation, part 2 (400 cases) is used for benchmarks in Novel Identity Rendering. The following information is available currently:

Part 1 Part 2 Part 3 Part 4
Actors 37 116 103 115
Motions Sequences 39 400 309 345
RGB Cameras
(with known camera matrix and color calibration parameters)
 60
Kinect Depth Cameras
(with known camera matrix)
 8
Image Dimension
(height x weight)
 cameraID in [0, 47]: 2448 x 2048
cameraID in [48, 59]: 4096 x 3000
depth_cameraID in [0, 7]: 576 x 640
Total RGB Frames
(60 views)
 513,000 4,999,500 4,171,500 4,657,500
Total Kinect Depth Frames
(8 views)
 68,400 666,600 556,220 621,000
Total Kinect Mask Frames
(8 views)
 68,400 666,600 556,220 621,000
2D keypoints
(60 views)
 513,000 4,999,500 / /
3D keypoints 8,550 83,325 / /
Matting
(60 views)
 513,000 4,999,500 / /
SMPLx 8,550 83,325 / /
Total storage ~192GB ~2.2T ~1.65T ~1.7T

For ten cases metioned in 4K4D, eight of them are NOT part of the official DNA-Rendering dataset. However, in order to make fair comparison in the research community, we have decided to release the data. The total storage takes around 40GB.

Timeline


More data will be released soon. Please stay tuned.

Main Data Format

Data is stored in SMC file through H5 format and each file is named as "{actorID}_{performanceID}.smc".
Structure of SMC file: 
{
    "attrs": {}        ### meta attributes
    "Camera_5mp": {}   ### attributes and images of 48 5MP cameras
    "Camera_12mp": {}  ### attributes and images of 12 12MP cameras
}
"attrs" stores actor's information and filename of textual description. We captured several motion sequences for each actor, "performance_id" indicates the index number of the corresponding actor's motion sequences. 
"attrs": {
    "actor_id"            :int
    "performance_id"      :int
    "age"                 :int
    "gender"              :str   ### ["male", "female"]
    "height"              :int
    "weight"              :int
    "ethnicity"           :str   ### ["yellow", "white", "black", "brown"]
    "text_label"          :str,  ### json filename, format is {actorID}_{performanceID}.json
}

We provide textual descriptions for each case, including three main contents: cloth of the person, interactive objects in the scene and action of the sequence. To fill the gap in Chinese annotations and enlarge the language categories in the text-to-2D/text-to-3D research area, we provide content in both Chinese and English.

  • Cloth: We provide descriptions of the type, material and texture of both tops and bottoms. For the shoes, color and type of the instance are provided, *e.g.*, black cloth shoes. In addition, if the person is wearing special costumes, information about the costumes is provided.
  • Interactive objects: We provide descriptions of interactive objects in the scene by the format of *list*, length of which is the number of objects.
  • Action: We provide descriptions of the overall action of the sequence by the format of *list*, including two versions provided by different developers.

Structure of {actorID}_{performanceID}.json file:

{
  "Cloth-Description-Chinese":                    # descriptions in Chinese of clothes of the person
   {
     'Clothing Type'      :
      {
        'Tops':
        'Bottoms':
      }
     'Clothing Material'  :
      {
        'Tops':
        'Bottoms':
      }
     'Clothing Texture'   :
      {
        'Tops':
        'Bottoms':
      }
      'Shoes'             :
      'Special Costumes'  :
   }
  "Cloth-Description-English":                    # descriptions in English of clothes of the person
   {
     ...
   }
  'Interactive-Objects-Chinese'   :   list[str]   # descriptions in Chinese of interactive objects in the scene
  'Interactive-Objects-English'   :   list[str]   # descriptions in English of interactive objects in the scene
  'Overall-Action-Chinese'        :   list[str]   # descriptions in Chinese of actions of the person
  'Overall-Action-English'        :   list[str]   # descriptions in English of actions of the person
}
An example is shown below
{
  "Cloth-Description-Chinese": {
    "Shoes": "白色绣花鞋",
    "Special Costumes": "汉服",
    "Clothing Type": {
      "Tops": "汉服",
      "Bottoms": "汉服"
    },
    "Clothing Material": {
      "Tops": "薄纱",
      "Bottoms": "薄纱"
    },
    "Clothing Texture": {
      "Tops": "印花",
      "Bottoms": "印花"
    }
  },
  "Cloth-Description-English": {
    "Shoes": "white embroidered shoes",
    "Special Costumes": "hanfu",
    "Clothing Type": {
      "Tops": "hanfu",
      "Bottoms": "hanfu"
    },
    "Clothing Material": {
      "Tops": "chiffon",
      "Bottoms": "chiffon"
    },
    "Clothing Texture": {
      "Tops": "printing",
      "Bottoms": "printing"
    }
  },
  "Interactive-Objects-Chinese": [
    "扇子"
  ],
  "Interactive-Objects-English": [
    "fan"
  ],
  "Overall-Action-Chinese": [
    "一个身穿古代服装的亚洲女子正在手拿扇子跳舞",
    "一个身着淡蓝色古装的亚洲女性正在用蝴蝶扇子做舞蹈动作"
  ],
  "Overall-Action-English": [
    "An Asian woman wearing a light blue antique costume is performing dance movements with a butterfly fan",
    "An Asian woman dressed in light blue antique attire performs dance movements with a pink butterfly fan"
  ]
}
"Camera_5mp" stores attributes of 5MP cameras and images of different views. "Camera_12mp" shares the same structure, while it stores attributes of 12MP cameras and higher quality images. 
"Camera_5mp": {
    "attrs": {
        "num_device"       :int
        "num_frame":       :int
        "resolution"       :(int, int) ### stored as (height, weight)
    },
    "cameraID": {
        "color":{
            "frameID":     :uint8  ### encoded byte array
        }
    }
}

Annotations Format

Annotation is stored in SMC file through H5 format and each file is named as "{actorID}_{performanceID}_annots.smc".
Structure of SMC file: 
{
    "Camera_Parameter": {                   # camera parameters defined in opencv coordination                     
      "cameraID": {                         
          "K"                             : # camera intrinsics, double (3, 3)                   
          "D"                             : # camera distortion, double (5,  )
          "RT"                            : # matrix that transforms from camera space to world space, double (4,4)                  
          "Color_Calibration"             : # calibration parameters in BGR format, double (3,3)
      }
    }
    "Mask": {                               : # foreground segmentation results
      "cameraID"                          : # uint8 (H, W)
    }
    "Keypoints_2D": {                       : # 2D keypoints of the person
      "cameraID"                          : # double (frame_num,joint_num,3)
    }                         
    "Keypoints_3D"                        : # 3D keypoints of the person, double (frame_num,joint_num,3)
    "SMPLx": {
      "beta"                              : # shape parameters of the person, double (1, 10)
      "full_pose"                         : # pose parameters of the person, double (frame_num, joint_num, 3)
      ...                                 : # other parameters
    }
}

Kinect Data Format

Kinect data is stored in SMC file through H5 format and each file is named as "{actorID}_{performanceID}_kinect.smc".
Structure of Kinect Data SMC file: 
{
    "Calibration": {
      "Kinect": {                            # kinect camera parameters defined in opencv coordination                                  
        "cameraID": {                         
            "K"                             : # camera intrinsics, double (3, 3)                   
            "D"                             : # camera distortion, double (5,  )
            "RT"                            : # matrix that transforms from camera space to world space, double (4,4)                  
        }
      }
    }
    "Kinect": {
      "attrs": {
          "num_device"       :int
          "num_frame":       :int
          "resolution"       :(int, int) ### stored as (height, weight)
      },
      "cameraID": {
          "depth":{
              "frameID":     :uint16  ### encoded byte array
          },
          "mask":{
            "frameID":       :uint8  ### encoded byte array
          }
      }
}
Please check "smc_reader_full.py" in dna_rendering_sample_code.zip for your reference. 
python smc_reader_full.py \
    --smc_file [location of smc file]
Depth information captured by kinect camera is useful to generate point-clouds. Please check "gen_pcd_from_kinect.py" in dna_rendering_sample_code.zip for your reference. 
python gen_pcd_from_kinect.py \
  --smc_rgb_file [smc file stored rgb images] \
  --smc_annot_file [smc file stored rgb camera matrix] \
  --smc_kinect_file [smc file stored kinect data]

Citation

If you find our dataset useful, please cite the following paper:

@article{2023dnarendering,
      title={DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering},
      author={Wei Cheng and Ruixiang Chen and Wanqi Yin and Siming Fan and Keyu Chen and Honglin He and Huiwen Luo and Zhongang Cai and Jingbo Wang and Yang Gao and Zhengming Yu and Zhengyu Lin and Daxuan Ren and Lei Yang and Ziwei Liu and Chen Change Loy and Chen Qian and Wayne Wu and Dahua Lin and Bo Dai and Kwan-Yee Lin},
      journal   = {arXiv preprint},
      volume    = {arXiv:2307.10173},
      year    = {2023}
}