[{"content":" I build 3D vision systems that recover hidden shape, texture, and fit, and take them from research paper to product. I\u0026rsquo;m a research engineer in 3D computer vision. I reconstruct people, garments, and objects from ordinary images, including the parts the camera can\u0026rsquo;t see. At Digi-Clone I lead R\u0026amp;D for a 3D virtual try-on product, from estimating the body to simulating the cloth. Before that, I did my Ph.D. at UT Dallas with Dr. Xiaohu Guo on occlusion-aware 3D reconstruction from hand-object interactions; that work produced TexHOI, which won the CVM 2025 Best Paper Award. My name, Alakh, is Sanskrit for \u0026ldquo;that which cannot be perceived by the eye\u0026rdquo;, which suits someone who spends his days recovering hidden surfaces.\nResume Email me Best Paper\nCVM 2025\nFor TexHOI, published in IEEE TVCG. AIR 835\nJEE Advanced\nAll-India Rank, 2015. ~200\nCitations\nOn Google Scholar. Now # Senior Research Engineer, Digi-Clone · Sep 2025 – present\nI\u0026rsquo;m the sole R\u0026amp;D engineer on a 3D virtual try-on product, leading the R\u0026amp;D funded by the company\u0026rsquo;s NSF SBIR Phase I award; I built the product\u0026rsquo;s technical MVP under it.\nTwo photos → a 3D body. It fits a parametric body model from a front photo, a side photo, and tape measurements. Garments on bodies. It places sewing-pattern panels on the avatar and drapes them with cloth simulation, across size-graded women\u0026rsquo;s and men\u0026rsquo;s garments. Measured, not eyeballed. Across 15 garments, every draped seam closes within 8.6 mm. 13 of 15 garments have zero body penetration, and fabric stretch averages under 1.2×. Selected work # TexHOI IEEE TVCG (CVM 2025), Best Paper Award. Reconstructs the shape and true surface color of an unknown object from a single video of a hand handling it. Layered-Garment Net ACCV 2022, with Samsung Research America. From one photo of a person, rebuilds each layer of clothing as separate 3D surfaces that never pass through each other. Earlier work: synthetic-to-real object detection (NCVPRIPG 2017, 100+ citations) · GAN-based face de-identification (IEEE IEMTRONICS 2020) · broad learning for aerial change detection (IEMGraph 2018) · vision-based navigation for an autonomous underwater vehicle (IIT BHU). All research\nExperience # Senior Research Engineer 2025 – present Digi-Clone · remote (Atlanta) R\u0026D for a 3D virtual try-on product: body estimation from photos, garment placement, and cloth draping. Ph.D., Computer Science 2019 – 2025 UT Dallas · Computer Graphics \u0026amp; Animation Lab GPA 4.0. Advisor Dr. Xiaohu Guo. Occlusion-aware 3D reconstruction from human-object interactions. Teaching assistant for Computer Graphics, Animation \u0026 Gaming, Algorithms, AI, and Databases. Research Intern 2018 Samsung Research Institute · Bangalore On-device lip reading. Research Fellow 2017 IIT Gandhinagar Sim-to-real object detection. B.Tech (Hons.), Computer Science and Engineering 2015 – 2019 IIT (BHU) Varanasi Joined with JEE Advanced All-India Rank 835. News # Oct 2025: Defended my Ph.D. thesis, \u0026ldquo;Learning Complex Occlusion-Aware 3D Representations from Human-Object Interactions.\u0026rdquo; Sep 2025: Joined Digi-Clone as Senior Research Engineer, leading R\u0026amp;D for 3D virtual try-on. Aug 2025: Digi-Clone received an NSF SBIR Phase I award. Apr 2025: TexHOI won the Best Paper Award at CVM 2025. Apr 2025: Received the Betty and Gifford Johnson Travel Award, UT Dallas. Apr 2025: Passed my Ph.D. proposal. Earlier news, 2015 – 2025 Feb 2025: TexHOI accepted to CVM 2025 and recommended to IEEE TVCG. Sep 2022: Layered-Garment Net accepted to ACCV 2022. Sep 2020: Face de-identification paper (EPD-Net) published at IEEE IEMTRONICS 2020. Aug 2019: Started my Ph.D. in Computer Science at UT Dallas. Jul 2019: Broad Neural Network paper published in Springer AICS. May 2019: Graduated with a B.Tech (Hons.) in CSE from IIT (BHU) Varanasi. May 2018: Research internship at Samsung Research Institute, Bangalore, on lip reading. Dec 2017: Presented synthetic-to-real object detection at NCVPRIPG 2017, IIT Mandi. May 2017: Research fellowship at IIT Gandhinagar. 2016: Champion, Technex Robosoccer, IIT (BHU). Jul 2015: Joined IIT (BHU) Varanasi with JEE Advanced All-India Rank 835. ","date":"6 May 2025","externalUrl":null,"permalink":"/","section":"Alakh Aggarwal","summary":"","title":"Alakh Aggarwal","type":"page"},{"content":"I reconstruct people, garments and objects from ordinary images, including the parts the camera can\u0026rsquo;t see. Full list on Google Scholar.\n","date":"6 May 2025","externalUrl":null,"permalink":"/research/","section":"Research","summary":"","title":"Research","type":"research"},{"content":"TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenes\nAlakh Aggarwal, Ningna Wang, Xiaohu Guo\nIEEE Transactions on Visualization and Computer Graphics (CVM 2025), 2025. 🏆 Best Paper Award\nProject page Paper Code Talk IEEE Reconstructs the shape and true surface color of an unknown object from a single video of a hand handling it. It accounts for the shadows and reflected light the hand casts on the object.\nResults # TexHOI\u0026rsquo;s albedo avoids the hand shadows and lighting that other methods bake into the texture. Compared with HOLD and InvRender. (Fig. 7 in the paper.) A reconstructed object can be relit under new environments. (Fig. 9 in the paper.) Abstract # Reconstructing 3D models of dynamic, real-world objects with high-fidelity textures from monocular frame sequences has been a challenging problem in recent years. This difficulty stems from factors such as shadows, indirect illumination, and inaccurate object-pose estimations due to occluding hand-object interactions. To address these challenges, we propose a novel approach that predicts the hand’s impact on environmental visibility and indirect illumination on the object\u0026rsquo;s surface albedo. Our method first learns the geometry and low-fidelity texture of the object, hand, and background through composite rendering of radiance fields. Simultaneously, we optimize the hand and object poses to achieve accurate object-pose estimations. We then refine physics-based rendering parameters—including roughness, specularity, albedo, hand visibility, skin color reflections, and environmental illumination—to produce precise albedo, and accurate hand illumination and shadow regions. Our approach surpasses state-of-the-art methods in texture reconstruction and, to the best of our knowledge, is the first to account for hand-object interactions in object texture reconstruction.\nCitation # @article{aggarwal2025texhoi, title = {TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenes}, author = {Aggarwal, Alakh and Wang, Ningna and Guo, Xiaohu}, journal = {IEEE Transactions on Visualization and Computer Graphics}, year = {2025}, doi = {10.1109/TVCG.2025.3567276} } ","date":"6 May 2025","externalUrl":null,"permalink":"/research/texhoi/","section":"Research","summary":"IEEE TVCG (CVM 2025), Best Paper Award. Reconstructs the shape and true surface color of an unknown object from a single video of a hand handling it.","title":"TexHOI","type":"research"},{"content":"","date":"17 April 2023","externalUrl":null,"permalink":"/posts/","section":"Blog","summary":"","title":"Blog","type":"posts"},{"content":" The rendering equation for PhySG, along a view-direction $o$ from camera, is defined as follows: $$R(o) = \\int_{S^2} L(i) \\rho_o(i) max(0, i.n) di$$Rendering Equation Using Spherical Gaussians (SGs) to render the scene is discussed in the previous post.\nThe mathematical properties of SGs are discussed before.\nPhySG in equations # $$R(o) = \\int_{S^2} L(i) \\rho_o(i) max(0, i.n) di$$$$L(i) = \\sum_{j=1}^N G(i; p_j, \\lambda_j, \\mu_j)$$$$i.n = G(i; n, 0.0315, 32.7080) - 31.7003$$$$\\rho_o(i) = \\rho_d(i) + \\rho_s(i)$$$$\\rho_d(i) = \\frac{a(x)}{\\pi}, \\rho_s(i) = M_o(i)D(h)$$, where $h = \\frac{i + o}{\\|\\|i + o\\|\\|_2}$ $$\\\\M_o(i) = \\frac{F(o,i)S(o,i)}{\\pi(n.i)(n.o)}$$ $$\\\\F(o,i) = s + (1-s).2^{-(5.55473o.h+6.8316)(o.h)}$$ $$\\\\S(o,i) = \\frac{o.n}{o.n(1-k)+k}.\\frac{i.n}{i.n(1-k)+k}$$ where $k = \\frac{(R+1)^2}{8}$, $R$ is the roughness, $s$ is the specular reflectance $$\\\\D(h) = \\frac{1}{\\pi R^4}e^{-(cos^{-1}(h.n)/R)^2} \\approx G(h; n, \\frac{2}{R^4}, \\frac{1}{\\pi R^4})$$To calculate $M_o(i)$: $$\\\\ $$We can approximate $i \\approx 2(o.n)n - o$. $$\\\\ $$So $h=n$, or $F(o,i) = s + (1-s)2^{-(5.55473o.n+6.8316)(o.n)}$. $$\\\\ $$Also, $S(o,i) = \\frac{o.n}{o.n(1-k)+k}.\\frac{(2(o.n)n - o).n}{(2(o.n)n - o).n(1-k)+k}$. $$\\\\M_o(i) = \\frac{s + (1-s)2^{-(5.55473o.n+6.8316)(o.n)} . \\frac{o.n}{o.n(1-k)+k}.\\frac{2(o.n)n.n - o.n}{(2(o.n)n.n - o.n)(1-k)+k}}{\\pi(2(o.n)n.n - o.n)(n.o)}$$To calculate $D(h)$: $$\\\\ $$Note, $D()$ is a function of the half vector $h$. So, it must be spherically warped to be a function of the incident light direction $i$. $$\\\\D^*(i) = G(i; p_D, \\lambda_D, \\mu_D)$$ where $p_D$ is the lobe orientation of the SG, $\\lambda_D$ is the lobe sharpness of the SG, and $\\mu_D$ is the lobe amplitude of the SG.\nThe spherical warp transformation is defines as: $$\\\\i = \\psi(h) = 2(o.h)h - o$$ where $\\psi()$ is the spherical warp transformation.\nThis follows: $$\\\\p_D = \\psi(n) = 2(o.n)n - o$$ $$\\\\\\lambda_D = \\frac{\\frac{2}{R^4}}{\\tau(h)} = \\frac{2}{4|o.h|R^4}$$ $$\\\\\\mu_D = \\frac{1}{\\pi R^4}$$Therefore $$\\rho_s(i) = M_o(i)D^*(i) \\approx G(i; p_D, \\lambda_D, M_o(i)*\\mu_D)$$Feeding all into Equation 1, we get: $$R(o) = \\int_{S^2} \\sum_{j=1}^N G(i; p_j, \\lambda_j, \\mu_j)(\\frac{a(x)}{\\pi} + G(i; p_D, \\lambda_D, M_o(i)*\\mu_D))(G(i; n, 0.0315, 32.7080) - 31.7003) di$$$$\\\\= \\frac{a(x)}{\\pi}\\int_{S^2} \\sum_{j=1}^N G(i; p^1_j, \\lambda^1_j, \\mu^1_j)di - \\frac{a(x)}{\\pi}\\int_{S^2} \\sum_{j=1}^N G(i; p^2_j, \\lambda^2_j, \\mu^2_j)di$$ $$\\\\+\\int_{S^2} \\sum_{j=1}^N G(i; p^3_j, \\lambda^3_j, \\mu^3_j)di - \\int_{S^2} \\sum_{j=1}^N G(i; p^4_j, \\lambda^4_j, \\mu^4_j)di$$To summarize: $$ R(o) = Term_1 - Term_2 + Term_3 - Term_4$$ $$\\\\Term_1 = \\frac{a(x)}{\\pi}\\int_{S^2} \\sum_{j=1}^N G(i; p^1_j, \\lambda^1_j, \\mu^1_j)di$$ $$\\\\p^1_j = \\frac{\\lambda_j n + 0.0315 p_j}{\\lambda_j + 0.0315}$$ $$\\\\\\lambda^1_j = \\lambda_j + 0.0315$$ $$\\\\\\mu^1_j = \\mu_j * 32.7080$$ $$\\\\Term_1 = 32.7080 * \\frac{a(x)}{\\pi} * \\mu_j \\int_{S^2} \\sum_{j=1}^N e^{(\\lambda_j + 0.0315)(i.\\frac{\\lambda_j n + 0.0315 p_j}{\\lambda_j + 0.0315} - 1)}di$$ $$\\\\Term_2 = \\frac{a(x)}{\\pi}\\int_{S^2} \\sum_{j=1}^N G(i; p^2_j, \\lambda^2_j, \\mu^2_j)di$$ $$\\\\p^2_j = p_j$$ $$\\\\\\lambda^2_j = \\lambda_j$$ $$\\\\\\mu^2_j = \\mu_j * 31.7003$$ $$\\\\Term_2 = 31.7003 * \\frac{a(x)}{\\pi} * \\mu_j \\int_{S^2} \\sum_{j=1}^N e^{\\lambda_j(i.p_j - 1)}di$$ $$\\\\Term_3 = \\int_{S^2} \\sum_{j=1}^N G(i; p^3_j, \\lambda^3_j, \\mu^3_j)di$$ $$\\\\p^3_j = \\frac{\\lambda^1_j p_D + \\lambda_D p^1_j}{\\lambda^1_j + \\lambda_D} $$ $$= \\frac{(\\lambda_j + 0.0315) (2*(o.n)n-o) + \\frac{2}{4|o.h|R^4}\\frac{\\lambda_j n + 0.0315 p_j}{\\lambda_j + 0.0315}}{\\lambda_j + 0.0315 + \\frac{2}{4|o.h|R^4}}$$ $$\\\\\\lambda^3_j = \\lambda_j + 0.0315 + \\lambda_D = \\lambda_j + 0.0315 + \\frac{2}{4|o.h|R^4}$$ $$\\\\\\mu^3_j = \\mu_j * 32.7080 * M_o(i) * \\mu_D = \\mu_j * 32.7080 * \\frac{s + (1-s)2^{-(5.55473o.n+6.8316)(o.n)} . \\frac{o.n}{o.n(1-k)+k}.\\frac{2(o.n)n.n - o.n}{(2(o.n)n.n - o.n)(1-k)+k}}{\\pi(2(o.n)n.n - o.n)(n.o)} * \\frac{1}{\\pi R^4}$$ $$\\\\Term_3 = \\mu_j * 32.7080 * \\frac{s + (1-s)2^{-(5.55473o.n+6.8316)(o.n)} . \\frac{o.n}{o.n(1-k)+k}.\\frac{2(o.n)n.n - o.n}{(2(o.n)n.n - o.n)(1-k)+k}}{\\pi(2(o.n)n.n - o.n)(n.o)} * \\frac{1}{\\pi R^4} \\int_{S^2} \\sum_{j=1}^N e^{(\\lambda_j + 0.0315 + \\frac{2}{4|o.h|R^4})(i.\\frac{(\\lambda_j + 0.0315) (2*(o.n)n-o) + \\frac{2}{4|o.h|R^4}\\frac{\\lambda_j n + 0.0315 p_j}{\\lambda_j + 0.0315}}{\\lambda_j + 0.0315 + \\frac{2}{4|o.h|R^4}} - 1)}di$$ $$\\\\Term_4 = \\int_{S^2} \\sum_{j=1}^N G(i; p^4_j, \\lambda^4_j, \\mu^4_j)di$$ $$\\\\p^4_j = \\frac{\\lambda^2_j p_D + \\lambda_D p^2_j}{\\lambda^2_j + \\lambda_D}$$ $$= \\frac{\\lambda_j (2*(o.n)n-o) + \\frac{2}{4|o.h|R^4}p_j}{\\lambda_j + \\frac{2}{4|o.h|R^4}}$$ $$\\\\\\lambda^4_j = \\lambda_j + \\frac{2}{4|o.h|R^4}$$ $$\\\\\\mu^4_j = \\mu_j * 31.7003 * M_o(i) * \\mu_D = \\mu_j * 31.7003 * \\frac{s + (1-s)2^{-(5.55473o.n+6.8316)(o.n)} . \\frac{o.n}{o.n(1-k)+k}.\\frac{2(o.n)n.n - o.n}{(2(o.n)n.n - o.n)(1-k)+k}}{\\pi(2(o.n)n.n - o.n)(n.o)} * \\frac{1}{\\pi R^4}$$ $$\\\\Term_4 = \\mu_j * 31.7003 * \\frac{s + (1-s)2^{-(5.55473o.n+6.8316)(o.n)} . \\frac{o.n}{o.n(1-k)+k}.\\frac{2(o.n)n.n - o.n}{(2(o.n)n.n - o.n)(1-k)+k}}{\\pi(2(o.n)n.n - o.n)(n.o)} * \\frac{1}{\\pi R^4} \\int_{S^2} \\sum_{j=1}^N e^{(\\lambda_j + \\frac{2}{4|o.h|R^4})(i.\\frac{\\lambda_j (2*(o.n)n-o) + \\frac{2}{4|o.h|R^4}p_j}{\\lambda_j + \\frac{2}{4|o.h|R^4}} - 1)}di$$","date":"17 April 2023","externalUrl":null,"permalink":"/posts/physg-eq/","section":"Blog","summary":"","title":"PhySG in equations","type":"posts"},{"content":"","date":"17 April 2023","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"17 April 2023","externalUrl":null,"permalink":"/tags/tech/","section":"Tags","summary":"","title":"Tech","type":"tags"},{"content":" The rendering equation, along a view-direction $o$ from camera, is defined as follows:\n$$R(o) = \\int_{S^2} L(i) V(i) \\rho_o(i) max(0, i.n) di$$Rendering Equation This rendering equation gives the final rendered color for a surface point $x$, illuminated by one or more light sources.\nHere, $i$ is the incident light direction.\nSo, $L(i)$ defines direct illumination from light, $V(i)$ determines if the visibility of the light source, $\\rho_o(i)$ is the Bidirection Radial Distribution Function (BRDF) that determines the albedo, diffuse and specularity, and $i.n$ gives the cosine angle between the surface and incident light direction.\nNote, in this equation, we use $max(0, i.n)$ to only sample incident lights from positive hemisphere towards the surface normal.\nThe rendering process can be optimized by using Spherical Gaussians (SGs) instead of the standard process of sampling the incident light directions. This is because SGs are a closed form function of the dot product of the incident light direction and the lobe orientation. This means that we can directly compute the integral in the rendering equation using the SGs. However, sampling the incident light directions is time-consuming, and we can avoid this by using SGs.\nThe mathematical properties of SGs are discussed in the previous post.\nRendering using SGs # We can define environment map $L(i)$ as a mixture model of SGs. The environment map is a function of the incident light direction $i$. \\ $$L(i) = \\sum_{j=1}^N G(i; p_j, \\lambda_j, \\mu_j)$$where $p_j$ is the lobe orientation of the $j^{th}$ SG, $\\lambda_j$ is the lobe sharpness of the $j^{th}$ SG, and $\\mu_j$ is the lobe amplitude of the $j^{th}$ SG.\nSimilarly, by defining the BRDF $\\rho_o(i)$ and the cosine angle $i.n$ as a mixture model of SGs, we can compute the integral in the rendering equation using the SGs.\n$$\\rho_o(i) = \\frac{a}{\\pi} + \\rho_s(i)$$ where a is the albeto of the surface, and $\\rho_s(i)$ is the specular component of the BRDF.\nThus, the diffuse component of the BRDF is defined as follows: $$R_d(o) = \\sum_{j=1}^N\\frac{a}{\\pi} \\int_{S^2} G(i; p_j, \\lambda_j, \\mu_j) V(i) max(0, i.n) di$$Similarly, the specular component of the BRDF is defined as follows: $$R_s(o) = \\sum_{j=1}^N\\int_{S^2} G(i; p_j, \\lambda_j, \\mu_j) V(i) \\rho_s(i) max(0, i.n) di$$We can further define $\\rho_s(i)$ as SGs, and compute the integral using the SGs.\nAccording to the microfacet theory, the specular component of the BRDF is defined as follows: $$\\rho_s(i) = M_o(i)D(h)$$where $M_o(i)$ is the combination of microfacet shadowing and Fresnel reflection, and $D(h)$ is the microfacet distribution function.\nHere, $h$ is the half vector between the incident light direction $i$ and the view direction $o$. $$h = \\frac{i+o}{||i+o||}$$ $$\\\\M_o(i) = \\frac{F(o,i)S(o,i)}{\\pi(n.i)(n.o)}$$where $F(o,i)$ is the Fresnel reflection, $S(o,i)$ is the microfacet shadowing, and $n.i$ and $n.o$ are the cosine angle between the surface normal and the incident light direction and the view direction respectively.\nWe can define $D(h)$ as follows according the microfacet theory: $$D(h) = e^{-(arccos(h.n)/m)^2}$$where $m$ is the roughness of the surface.\nWe can approximate this NDF with a single SG lobe as follows: $$D(h) = G(h; n, 2/m^2, 1)$$Note, $D()$ is a function of the half vector $h$. So, it must be spherically warped to be a function of the incident light direction $i$. $$D^*(i) = G(i; p_D, \\lambda_D, \\mu_D)$$where $p_D$ is the lobe orientation of the SG, $\\lambda_D$ is the lobe sharpness of the SG, and $\\mu_D$ is the lobe amplitude of the SG.\nThe spherical warp transformation is defines as: $$i = \\psi(h) = 2(o.h)h - o$$where $\\psi()$ is the spherical warp transformation.\nThis follows: $$p_D = \\psi(n) = 2(o.n)n - o$$ $$\\lambda_D = \\frac{2/m^2}{\\tau(n)}$$ $$\\mu_D = 1$$Differential area of this warp, $\\tau(h)$, is the determinant of the Jacobian matrix of the spherical warp transformation. $$\\tau(h) = \\frac{\\partial i}{\\partial h} = \\frac{||\\frac{\\partial \\psi(h)}{\\partial \\theta} X \\frac{\\partial \\psi(h)}{\\partial \\phi}||}{||\\frac{\\partial h}{\\partial \\theta} X \\frac{\\partial h}{\\partial \\phi}||} = 4|h.o|$$where $h(\\theta, \\phi)$ is the spherical coordinates of the half vector $h$, i.e., $(\\sin \\theta \\cos \\phi, \\sin \\theta \\sin \\phi, \\cos \\theta)$.\nHence, $$\\rho_s(i) = M_o(i)D(h) = M_o(i)D^*(i)$$where $D^*(i)$ is the SG approximation of the microfacet distribution function.\nFurther simplification gives, $$\\rho_s(i) = M_o(i)G(i; p_D, \\lambda_D, \\mu_D) = G(i; p_D, \\lambda_D, M_o(i)*\\mu_D)$$where $p_D$ is the lobe orientation of the SG, $\\lambda_D$ is the lobe sharpness of the SG, and $\\mu_D$ is the lobe amplitude of the SG.\nThus, the specular component of the BRDF is defined as follows: $$R_s(o) = \\sum_{j=1}^N\\int_{S^2} \\sum_{j=1}^N V(i) G(i; p_D, \\lambda_D, M_o(i)*\\mu_D) max(0, i.n) di$$where $p_D$ is the lobe orientation of the SG, $\\lambda_D$ is the lobe sharpness of the SG, and $\\mu_D$ is the lobe amplitude of the SG.\nSimilarly, by emperical fitting methods, we can define $i.n$ as an SG: $$i.n = G(i; n, 0.0315, 32.7080) - 31.7003$$where $n$ is the surface normal.\nThus, the diffuse component of the BRDF is defined as follows: $$R_d(o) = \\frac{a}{\\pi}\\sum_{j=1}^N \\int_{S^{2+}} G(i; p_j, \\lambda_j, \\mu_j) V(i) G(i; n, 0.0315, 32.7080) di \\\\ - \\frac{a}{\\pi}*31.7003\\sum_{j=1}^N\\int_{S^{2+}} G(i; p_j, \\lambda_j, \\mu_j) V(i)di$$And the specular component of the BRDF is defined as follows: $$R_s(o) = \\sum_{j=1}^N\\int_{S^{2+}} G(i; p_j, \\lambda_j, \\mu_j) V(i) G(i; p_D, \\lambda_D, M_o(i)*\\mu_D) G(i; n, 0.0315, 32.7080) di \\\\ - 31.7003\\sum_{j=1}^N\\int_{S^{2+}} G(i; p_j, \\lambda_j, \\mu_j) V(i) G(i; p_D, \\lambda_D, M_o(i)*\\mu_D) di$$where, $S^{2+}$ is the upper hemisphere of the surface normal direction $i$.\nWe already discussed how to calculate $M_o(i)$ before.\nParameters to learn are as follows:\n$a$ is the albedo of the surface. $p_j$ is the lobe orientation of the $j^{th}$ Lighting SG. $\\lambda_j$ is the lobe sharpness of the $j^{th}$ Lighting SG. $\\mu_j$ is the lobe amplitude of the $j^{th}$ Lighting SG. $p_D$ is the lobe orientation of the Specular SG. $\\lambda_D$ is the lobe sharpness of the Specular SG. $\\mu_D$ is the lobe amplitude of the Specular SG. Visibility $V(i)$ needs to be calculated.\n","date":"1 February 2023","externalUrl":null,"permalink":"/posts/sg-render/","section":"Blog","summary":"","title":"Rendering using Spherical Gaussians","type":"posts"},{"content":" A spherical Gaussian is a function defined on the surface of a sphere. It is defined as follows:\n$$G(v; p, \\lambda, \\mu) = \\mu e^{\\lambda(p.v-1)}$$where $v \\in S^2$ is the input vector, $\\mu \\in R$ is the lobe amplitude, $\\lambda \\in R$ is the lobe sharpness, and $p \\in S^2$ is the lobe orientation.\nSpherical Gaussian The spherical Gaussian is a generalization of the Gaussian function, which is defined on the plane. The spherical Gaussian is a function of the dot product of the input vector and the lobe orientation. The lobe orientation is a vector that defines the direction of the lobe. The lobe sharpness is a parameter that controls the width of the lobe. The lobe amplitude is a parameter that controls the height of the lobe.\nThe total energy of the spherical Gaussian is 1. i.e., $$\\int_{S^2} G(v; p, \\lambda, \\mu) dv = 1$$ Some mathematical properties of spherical Gaussians # Product of 2 spherical Gaussians is a spherical Gaussian: # $$G(v; p, \\lambda, \\mu) G(v; p', \\lambda', \\mu') = G(v; p_m, \\lambda_m, \\mu_m)$$Proof: $$G(v; p, \\lambda, \\mu) G(v; p', \\lambda', \\mu') \\\\= \\mu \\mu' e^{\\lambda(p.v-1)} e^{\\lambda'(p'.v-1)} \\\\= \\mu \\mu' e^{(\\lambda p.v + \\lambda'p'.v) - (\\lambda + \\lambda')} \\\\= \\mu\\mu'e^{(\\lambda + \\lambda')(\\frac{\\lambda p + \\lambda'p'}{\\lambda + \\lambda'}.v - 1)}$$where $p_m = \\frac{\\lambda p + \\lambda'p'}{\\lambda + \\lambda'}$, $\\lambda_m = \\lambda + \\lambda'$ and $\\mu_m = \\mu\\mu'$\nMixture model of scattered spherical Gaussians: # $$F^*(v) = \\sum_{i=1}^N w_i G(v; p_i, \\lambda_i, \\mu_i)$$where $w_i$ is the weight of the $i^{th}$ spherical Gaussian, $p_i$ is the lobe orientation of the $i^{th}$ spherical Gaussian, $\\lambda_i$ is the lobe sharpness of the $i^{th}$ spherical Gaussian, and $\\mu_i$ is the lobe amplitude of the $i^{th}$ spherical Gaussian.\nAn SG mixture model is closed under rotations. $$R \\circ F^*(v) = \\sum_{i=1}^N w_i G(v; R \\circ p_i, \\lambda_i, \\mu_i)$$where $R$ is a rotation matrix.\nSimilarly, multiplication of all SGs of a mixture model with a constant. $$\\sum_{i=1}^N c*w_i G(v; p_i, \\lambda_i, \\mu_i) = c*F^*(v) = \\sum_{i=1}^N w_i G(v; p_i, \\lambda_i, c * \\mu_i)$$where $c$ is a constant.\n","date":"1 February 2023","externalUrl":null,"permalink":"/posts/sg/","section":"Blog","summary":"","title":"Spherical Gaussians","type":"posts"},{"content":"Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image\nAlakh Aggarwal, Jikai Wang, Steven Hogue, Saifeng Ni, Madhukar Budagavi, Xiaohu Guo\nAsian Conference on Computer Vision (ACCV), 2022. With Samsung Research America.\nPaper PDF Code Talk From one photo of a person, it rebuilds each layer of clothing (a shirt under a jacket, over trousers) as separate 3D surfaces that never pass through each other.\nTalk # Method # From a single image, LGN predicts a signed distance field for each garment layer plus a garment indication field that keeps outer layers covering inner ones without intersecting. Results # Compared with BCNet and SMPLicit, LGN keeps the inner shirt under the jacket without the layers passing through each other. BCNet cannot model a jacket over a shirt. Multi-view reconstructions on three people: LGN (left), SMPLicit (middle), BCNet (right). Abstract # Recent research works have focused on generating human models and garments from their 2D images. However, state-of-the-art researches focus either on only a single layer of the garment on a human model or on generating multiple garment layers without any guarantee of the intersection-free geometric relationship between them. In reality, people wear multiple layers of garments in their daily life, where an inner layer of garment could be partially covered by an outer one. In this paper, we try to address this multi-layer modeling problem and propose the Layered-Garment Net (LGN) that is capable of generating intersection-free multiple layers of garments defined by implicit function fields over the body surface, given the person\u0026rsquo;s near front-view image. With a special design of garment indication fields (GIF), we can enforce an implicit covering relationship between the signed distance fields (SDF) of different layers to avoid self-intersections among different garment surfaces and the human body. Experiments demonstrate the strength of our proposed LGN framework in generating multi-layer garments as compared to state-of-the-art methods. To the best of our knowledge, LGN is the first research work to generate intersection-free multiple layers of garments on the human body from a single image.\nCitation # @inproceedings{aggarwal2022lgn, title = {Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image}, author = {Aggarwal, Alakh and Wang, Jikai and Hogue, Steven and Ni, Saifeng and Budagavi, Madhukar and Guo, Xiaohu}, booktitle = {Proceedings of the Asian Conference on Computer Vision (ACCV)}, pages = {3000--3017}, year = {2022} } ","date":"4 December 2022","externalUrl":null,"permalink":"/research/lgn/","section":"Research","summary":"ACCV 2022, with Samsung Research America. From one photo of a person, rebuilds each layer of clothing as separate 3D surfaces that never pass through each other.","title":"Layered-Garment Net","type":"research"},{"content":"EPD-Net: A GAN-based Architecture for Face De-identification from Images\nAlakh Aggarwal, Rishika Rathore, Pratik Chattopadhyay, Lipo Wang\nIEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), 2020.\nPaper PDF A GAN that hides who is in a face image while keeping the expression and other non-identifying information, for sharing surveillance footage without exposing identities.\nResults # Top: faces from the RaFD dataset. Bottom: EPD-Net\u0026rsquo;s de-identified outputs, which change who the person appears to be while keeping the expression. (Fig. 3 in the paper.) Method # A structure-preserving GAN (U-Net generator, PatchGAN discriminator with SSIM loss) trained against two verifiers: one pushes identity away from the input, the other keeps the emotion. (Fig. 2 in the paper.) Abstract # Nowadays huge amount of crowd data captured by surveillance cameras gets shared publicly in the form of images or videos through television or the internet. Although many of these videos are meant to provide public security, they also lead to a widespread concern towards privacy protection, since a lot of personal information about subjects gets revealed through this video/image data. Hence, \u0026ldquo;de-identifying\u0026rdquo; people (i.e., obscuring identity information) captured by the surveillance cameras is of utmost importance for providing privacy along with security. Traditional identity obfuscation techniques such as blurring, warping, and filtering lead to the loss of vital non-biometric information. More recent k-same-based as well as generative model-based de-identification techniques eliminate the above problem to a certain extent. Still, the visual quality of the generated images produced by these methods is not realistic. Also, these approaches are unable to maintain structural integrity and cannot preserve the required non-biometric information at a high resolution. As an improvement, in this paper, we propose a new network termed as EPD-Net and train it with suitable loss functions to maximize the emotion similarity and minimize the identity similarity. Experimental results verify the effectiveness of our approach and its superiority over other popular face de-identification techniques.\nCitation # @inproceedings{aggarwal2020epdnet, title = {EPD-Net: A GAN-based Architecture for Face De-identification from Images}, author = {Aggarwal, Alakh and Rathore, Rishika and Chattopadhyay, Pratik and Wang, Lipo}, booktitle = {2020 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS)}, pages = {1--7}, year = {2020} } ","date":"9 September 2020","externalUrl":null,"permalink":"/research/epd-net/","section":"Research","summary":"IEEE IEMTRONICS 2020. A GAN that hides who is in a face image while keeping expression and other non-identifying information.","title":"EPD-Net","type":"research"},{"content":" This post mainly covers following works in brief:\nSMPL videoavatars Octopus DeepHuman SMPL (Skinned Multi-Person Linear Model) # Trained parameters of an SMPL default body are as follows:\nT ==\u0026gt; Mesh Vertices W ==\u0026gt; Blending Weights S ==\u0026gt; Shape Displacement Vectors P ==\u0026gt; Pose Displacement Vectors J ==\u0026gt; Joint Transformation Matrix Given two parameters, $\\theta$ for pose of human body and $\\beta$ for shape of a subject, default human body is transformed to subject\u0026rsquo;s geometric characteristics and a particular pose as encoded by $\\theta$\n","date":"15 June 2020","externalUrl":null,"permalink":"/posts/3d-human-rec/","section":"Blog","summary":"","title":"Deep Learning based 3D Human Body Reconstruction","type":"posts"},{"content":"I got the idea of this blog post when a friend of mine shared an answer on Quora. In this answer, Miguel Oliveira explains how the Machine Learning algorithm performed effectively, that is, took less time on increasing training batch size just because of changing to a more efficient data structure for the problem. This concept is usually overlooked by many people in the industries, making them wondering if they should actually care about efficient algrithms.\nIn major areas, including research, Deep Learning is used as a black box, without actually realizing the implications of optimization the uderlying algorithm for the network architecture. Forget that, some people do not even understand the network architecture. I came across a blog post of Adit Deshpande few years ago that lists and discusses about the 9 Deep Learning Papers you need to know. It discusses the 9 Deep Learning papers that lead to a strong foundation of the current researches and sucesses of Deep Neural Networks, and I find that algorithm optimization or using a better data structure strikingly improves the performance manifolds.\nIf we look into each of the recent breakthrough papers, most of the papers talk about optimization of algorithm(s) that lead to the efficient performance of the neural networks. The first paper, AlexNet introduces parallel computations, along with a result on a very large number of data, ImageNet. The Accuracy Score obtained was significantly high because of the use of parallel computation. The onset of Deep Neural Networks was also obvious with this work, because the results were achievable sooner than before due to the fast processing because of parallel processing, along with better computation facilities, obviously.\nAlexNet Although the increase in the computation power saw a linear growth, the efficiency of the data structures and algorithms underlying the network architectures saw an exponential growth. This led to a very rapid development of state-off-the-art Deep Learning algorithms in many areas. Although further advances in classification networks saw the bright light only because of parameter update and efficient visualization strategies or stacking of layers efficiently, as in VGGNet, GoogLeNet, Microsoft ResNet, etc. This made the underlying algorithms at the implementation level more complex.\nHowever, in this post, I would actually like to discuss the trilogy works of Ross Girshik et. al. on Object Detection and Localization, and how the simple improvements in the optimizations of the underlying algorithms led to a significant improvements in performance.\nProblem: Given an image, detect an object from the list of objects.\nBasic initution - Divide and Conquer # Input image: Input Divide the image into parts by horizontal and vertical cut. (Divide) Divide Consider each part. (Conquer) Pass the part image through a classification network and get the output score, along with the class predictions. Combine the part regions according to the output score and same class prediction (Combine). Regress the bounding box coordinates. Predict R-CNN # Find region proposals (~2K). Use Selective Search. For each region, Pass it through a classification network and get the output score, along with the class predictions. Pass the region through a fully convolutional network to get the feature maps. Pass the output feature map to the fully connected network to get the classification results. Regress the bounding box coordinates. Fast R-CNN # Pass the input image through a fully convolutional network to get the feature map. Find the region proposals on the feature map using Selective Search. For each region on the feature map, find the classification results. Regress the bounding box coordinates. Faster R-CNN # Pass the input image through a fully convolutional network to get the feature map. Find the region proposals on the feature map using Region Proposal Network. For each region on the feature map, find the classification results. Regress the bounding box coordinates. YOLO # Resize image. Pass the image through a fully convolutional neural network to get a fixed size output sXsXmX(c+5). Flatten the last two dimensions to get an output of dimension sXsX(c+5).m Combine the grids to get the results. Having a good GPU and computation power is not enough. We do not have infinite computability. Many researchers still find better, more optimal solutions, and it greatly helps.\n","date":"15 June 2020","externalUrl":null,"permalink":"/posts/gpu-efficiency/","section":"Blog","summary":"","title":"Why do I need efficiency when I have a great GPU?","type":"posts"},{"content":"","date":"14 June 2020","externalUrl":null,"permalink":"/tags/intro/","section":"Tags","summary":"","title":"Intro","type":"tags"},{"content":"Welcome to my Website! I am really glad that you are here, and hope that you can make the most out of it. I will be updating my profile and information and maintain it, along with writing about latest research in my field. I hope this keeps me up-to-date with latest works, and simultaneously, can be helpful for others.\nHope you like it! Shoot your comments, suggestions and thoughts at alakh.aggarwal@utdallas.edu.\n","date":"14 June 2020","externalUrl":null,"permalink":"/posts/welcome/","section":"Blog","summary":"","title":"Welcome to my Website!","type":"posts"},{"content":"Broad Neural Network for Change Detection in Aerial Images\nShailesh Shrivastava, Alakh Aggarwal, Pratik Chattopadhyay\nEmerging Technology in Modelling and Graphics (IEMGraph 2018), Springer AISC vol. 937, 2020.\nPaper arXiv My part was the classifier: a Broad Learning System that keeps adding layers of enhancement nodes, each compressed with a sparse autoencoder, and stops once the cross-validated F-score stops improving. My co-author Shailesh Shrivastava applied it to detecting which pixels changed between two aerial images of the same place, where it beat a multilayer perceptron on F-score.\nMethod # The Broad Learning System: new layers of enhancement nodes are added one at a time, then the input and all enhancement layers are concatenated and connected to the output by a single weight matrix W. (Fig. 2 in the paper.) Results # Top: two aerial images of the same place taken at different times. Bottom: the ground-truth change mask (left) and the map predicted by the Broad Learning classifier (right), with a 10:1 class imbalance. (Fig. 3 in the paper.) Abstract # A change detection system takes as input two images of a region captured at two different times and predicts which pixels in the region have undergone change over the time period. Since pixel-based analysis can be erroneous due to noise, illumination difference and other factors, contextual information is usually used to determine the class of a pixel (changed or not). This contextual information is taken into account by considering a pixel of the difference image along with its neighborhood. With the help of ground truth information, the labeled patterns are generated. Finally, Broad Learning classifier is used to get prediction about the class of each pixel. Results show that Broad Learning can classify the data set with a significantly higher F-score than that of Multilayer Perceptron. Performance comparison has also been made with other popular classifiers, namely Multilayer Perceptron and Random Forest.\nCitation # @inproceedings{shrivastava2020broad, title = {Broad Neural Network for Change Detection in Aerial Images}, author = {Shrivastava, Shailesh and Aggarwal, Alakh and Chattopadhyay, Pratik}, booktitle = {Emerging Technology in Modelling and Graphics}, series = {Advances in Intelligent Systems and Computing}, volume = {937}, publisher = {Springer}, year = {2020}, doi = {10.1007/978-981-13-7403-6_31} } ","date":"17 July 2019","externalUrl":null,"permalink":"/research/broad-learning-change-detection/","section":"Research","summary":"IEMGraph 2018, Springer. My part: a Broad Learning System that keeps adding its own enhancement layers until accuracy stops improving, applied to finding changed pixels in aerial images.","title":"Broad Neural Network for Change Detection","type":"research"},{"content":"Transfer Learning by Finetuning Pretrained CNNs Entirely with Synthetic Images\nParam S. Rajpura, Alakh Aggarwal, Manik Goyal, Sanchit Gupta, Jonti Talukdar, Hristo Bojinov, Ravi S. Hegde\nNCVPRIPG 2017, Springer CCIS vol. 841, 2018. 100+ citations.\nPaper arXiv Fine-tuning a pretrained object detector only on rendered images of grocery products more than doubled its accuracy on real refrigerator photos (mAP 24.15 → 52.59).\nMethod # 3D models of products and a refrigerator are rendered into annotated scenes, and a pretrained detector is fine-tuned on them alone. (Fig. 1 in the paper.) (a) Some of the 3D product models. (b) Rendered training scenes with varied textures, materials, packing and camera angles; photorealism was not the goal. (c) Real refrigerator photos used for testing. (Fig. 2 in the paper.) Results # Detections on real photos. Top row: every object of interest is found. Middle row: some objects are missed. Bottom row: the detector reports objects that are not there. (Fig. 8 in the paper.) Abstract # We show that finetuning pretrained CNNs entirely on synthetic images is an effective strategy to achieve transfer learning. We apply this strategy for detecting packaged food products clustered in refrigerator scenes. A CNN pretrained on the COCO dataset and fine-tuned with our 4000 synthetic images achieves mean average precision (mAP @ 0.5-IOU) of 52.59 on a test set of real images (150 distinct products as objects of interest and 25 distractor objects) in comparison to a value of 24.15 achieved without such finetuning. The synthetic images were rendered with freely available 3D models with variations in parameters like color, texture and viewpoint without a high emphasis on photorealism. We analyze factors like training data set size, cue variances, 3D model dictionary size and network architecture for their influence on the transfer learning performance. Additionally, training strategies like fine-tuning with selected layers and early stopping which affect transfer learning from synthetic scenes to real scenes were explored. This approach is promising in scenarios where limited training data is available.\nCitation # @inproceedings{rajpura2018transfer, title = {Transfer Learning by Finetuning Pretrained CNNs Entirely with Synthetic Images}, author = {Rajpura, Param S. and Aggarwal, Alakh and Goyal, Manik and Gupta, Sanchit and Talukdar, Jonti and Bojinov, Hristo and Hegde, Ravi S.}, booktitle = {Computer Vision, Pattern Recognition, Image Processing, and Graphics (NCVPRIPG 2017)}, series = {Communications in Computer and Information Science}, volume = {841}, pages = {517--528}, publisher = {Springer}, year = {2018}, doi = {10.1007/978-981-13-0020-2_45} } ","date":"26 April 2018","externalUrl":null,"permalink":"/research/synthetic-transfer-learning/","section":"Research","summary":"NCVPRIPG 2017, Springer, 100+ citations. Fine-tuning an object detector only on rendered images more than doubled its accuracy on real photos.","title":"Transfer Learning with Synthetic Images","type":"research"}]