Expressivity

Research at our chair

• Expressivity of transformer architecture
  Sumformer: Universal Approximation for Efficient Transformers. https://proceedings.mlr.press/v221/alberti23a/alberti23a.pdf
• Expressivity of Spiking Neural Networks
  Expressivity of Spiking Neural Networks https://arxiv.org/abs/2308.08218
• Analyzing approximation rates of deep (ReLU) networks
  Error bounds for approximations with deep ReLU neural networks in W^{s,p} norms.https://arxiv.org/abs/1902.07896

General references

• Function approximation with deep (ReLU) neural networks
  Deep Neural Network Approximation Theory https://arxiv.org/pdf/1901.02220
• Expressivity of transformer architecture
  Sumformer: Universal Approximation for Efficient Transformers. https://proceedings.mlr.press/v221/alberti23a/alberti23a.pdf
  Are transformers universal approximators of sequence-to-sequence functions? https://arxiv.org/pdf/1912.10077
  Representational Strengths and Limitations of Transformers.https://arxiv.org/pdf/2306.02896
• Expressivity of neural networks in terms of the number of linear regions
  On the Number of Linear Regions of Deep Neural Networks.https://arxiv.org/pdf/1402.1869
  Bounding and Counting Linear Regions of Deep Neural Networks. https://arxiv.org/abs/1711.02114

Contact

Do you have questions about our research in this area?

Please do not hesitate to contact us directly. Feel free to write an e-mail to Manjot Singh, one of our PhD students in the field of Expressivity.

Inquiries from students are very welcome!