$ℓ_1$ based sparse regularization plays a central role in compressive sensing and image processing. In this paper, we propose $ℓ_1$DecNet, as an unfolded network derived from a variational decomposition model, which incorporates $ℓ_1$ related sparse regularizations and is solved by a non-standard scaled alternating direction method of multipliers. $ℓ_1$DecNet effectively separates a spatially sparse feature and a learned spatially dense feature from an input image, and thus helps the subsequent spatially sparse feature related operations. Based on this, we develop $ℓ_1$DecNet+, a learnable architecture framework consisting of our $ℓ_1$DecNet and a segmentation module which operates over extracted sparse features instead of original images. This architecture combines well the benefits of mathematical modeling and data-driven approaches. To our best knowledge, this is the first study to incorporate mathematical image prior into feature extraction in segmentation network structures. Moreover, our $ℓ_1$DecNet+ framework can be easily extended to 3D case. We evaluate the effectiveness of $ℓ_1$DecNet+ on two commonly encountered sparse segmentation tasks: retinal vessel segmentation in medical image processing and pavement crack detection in industrial abnormality identification. Experimental results on different datasets demonstrate that, our $ℓ_1$DecNet+ architecture with various lightweight segmentation modules can achieve equal or better performance than their enlarged versions respectively. This leads to especially practical advantages on resource-limited devices.