Lightweight Room Layout Estimation

2022 · ICCAS — lightweight 3D room layout estimation from a single panorama.

ICCAS 2022 · Busan Lightweight Deep Learning Neural Architecture Search

22nd International Conference on Control, Automation and Systems (ICCAS), BEXCO, Busan · Nov 2022 · Indexed in IEEE Xplore

Dayoung Kil, Seong-heum Kim · VIP Lab, Soongsil University

Can a phone-sized network rebuild the 3D layout of a room from a single panorama?

We make HorizonNet lightweight: its ResNet backbone is replaced by a searched MnasNet, and its LSTM by a GRU. The result runs on under half the parameters, with almost no loss in layout accuracy.

IEEE Xplore View code on GitHub

From a single panorama to a full 3D room layout.

The problem

Sharing the inside of a home as a single photo or panorama is commonplace now — but a 2D image distorts the real size and proportions of a 3D space. Room layout estimation recovers the true 3D structure (floor, ceiling, walls) from one image, which is useful for architects, interior design, and AR.

The catch: state-of-the-art models like HorizonNet are heavy, and camera ISPs / embedded platforms have a tight compute budget.

The goal: keep HorizonNet’s layout quality, but make it light enough for on-device, low-power use.

Approach

HorizonNet recovers a layout in three stages. We leave the pipeline intact and slim down only the middle one — the feature-extraction network.

Pre-processing

align the panorama · detect vanishing points

Feature extraction

predict 1D ceiling / floor / wall boundaries

we make this lightweight
Post-processing

lift to 3D under the Manhattan-world assumption

The feature extractor in HorizonNet is ResNet-50 + LSTM. We replace both halves with lighter modules and search the configuration instead of hand-tuning it:

Stage HorizonNet (baseline) Ours (lightweight)
Backbone ResNet-50 MnasNet — platform-aware NAS
Sequence model LSTM (2 states, 3 gates) GRU (1 state, 2 gates)
Hyperparameters fixed sampling-based search

Why these swaps

ResNet-50 → MnasNet

MnasNet decomposes the network into blocks over a factorized hierarchical search space — each block can differ, but layers inside a block share structure, so the search space stays small and mobile-friendly.

LSTM → GRU

A GRU merges the LSTM's input / forget gates into one update gate, and its cell / hidden states into a single state — fewer parameters for the same sequence modeling.

Searching the backbone

We tune the 6 inverted-residual blocks with sampling-based optimization. Mirroring ResNet’s 256·512·1024·2048 blew up the parameter count, so we use 128 · 256 · 512 · 1024 · 36 · 24 out-channels, and assign fewer repeats to the wide blocks — cutting FLOPs and parameters at almost no accuracy cost.

Factorized hierarchical search space of our MnasNet backbone. Each of the 6 blocks is tuned by sampling-based optimization over kernel size, stride, expansion ratio, repeat count, and output channels.

Results

Trained on a Stanford2D3D + PanoContext mix (817 train / 79 val / 166 test, 300 epochs), and validated on real RICOH THETA Z1 panoramas.

Metric HorizonNet (ResNet-50 + LSTM) Ours (MnasNet + GRU)
Parameters 81.6 M 37.6 M (−54%)
2D IoU 87.07 85.07
3D IoU 84.53 81.89
MSE 0.18 0.21

−54%

parameters (81.6M → 37.6M)

−2.0

2D IoU points only

≈ same

qualitative 3D layout
Qualitative comparison on real RICOH THETA panoramas: (a) input, (b) 1D layout from our MnasNet-GRU, (c) our 3D estimate, (d) the original HorizonNet (ResNet-50 + LSTM). The lightweight model is visually indistinguishable from the full one.

On real THETA panoramas, the lightweight model’s 3D reconstructions show no significant visual difference from the original HorizonNet.

Ablation — where the savings come from

Model Parameters FLOPs MSE
ResNet-50 + LSTM (baseline) 81.6 M 71.83 0.18
MnasNet + LSTM 40.4 M 59.19 0.23
MnasNet + GRU (ours) 37.6 M 58.48 0.21
  • MnasNet does the heavy lifting — it alone roughly halves the parameters (81.6M → 40.4M) and cuts FLOPs (71.83 → 59.19).
  • GRU trims a bit more (40.4M → 37.6M) and, interestingly, improves MSE (0.23 → 0.21): once the model is right-sized, the extra under-trained parameters hurt more than they helped.

Takeaways

  • A searched MobileNet-style backbone + GRU makes panoramic room-layout estimation embedded-friendly with under half the parameters.
  • The accuracy cost is small (≈2 IoU points), and qualitatively the 3D layouts are indistinguishable from the full model.

Future work: add the Structured3D dataset, and use reinforcement learning to search the remaining inverted-residual hyperparameters (kernel size, expansion ratio, stride).

Details

HorizonNet MnasNet (NAS) GRU Stanford2D3D · PanoContext RICOH THETA Z1

Authors: Dayoung Kil, Seong-heum Kim · VIP Lab, Soongsil University.
Supported by the National Research Foundation of Korea (MSIT), Grant NRF-2021R1G1A1009828.