Images to Code: Multi-Modal GPT-4 in Converting Designs to Code

A Practical Look into Generating Dart Code with GPT-4 from Visual Inputs.

Among the most intriguing developments in the field of AI coding is the advent of multi-modal AI systems, particularly the GPT-4 model developed by OpenAI. This model’s ability to interpret various forms of data, including textual and visual inputs, and generate relevant and sophisticated outputs, marks a significant leap in AI capabilities. In this article, we delve into an innovative application of multi-modal GPT-4: generating Dart code for Flutter applications based on images.

Our exploration revolves around a series of experiments where multi-modal GPT-4, accessed through ChatGPT, is provided with images and tasked with generating Dart code. These experiments are not just about converting visuals into code; they aim to push the boundaries of AI-assisted development. We will examine various scenarios: from using design mockups commonly found on platforms like Dribbble to generate front-end pages, to interpreting specific aspects of design reference images such as background effects or header areas. We even venture into more abstract realms, like translating the mood or ambiance captured in an image into frontend code that resonates with those feelings.

These experiments represent a step towards a future where the line between visual design and code becomes increasingly blurred, where ideas can transition into tangible products with unprecedented speed and efficiency. This article aims to shed light on the possibilities and challenges of integrating AI into Flutter app development, providing insights and inspirations for developers and enthusiasts alike.

Experiment 1 — Design Mockups to Front-End Pages

This first experiment covers what might be the most immediate and obvious use case that comes to mind: testing the capabilities of multi-modal GPT-4 in translating a design mockup for a mobile app into functional Dart code for Flutter applications. So, imagine that you, as a Flutter devloper, have been provided with a design mockup from a design team, or just from a site like Dribbble. You are tasked with creating a Flutter screen to match the design. Could you simply use ChatGPT for this task and go home for the weekend early?

Selecting a Design Mockup

For this experiment, let’s choose a relatively simple UI for a login page, something that many applications will include.

Generate Some Code

Alright, let’s give this a try. For this example, let’s try a simple prompt asking ChatGPT (using GPT-4) to create a widget that will match the design in the image above. The image is attached to the prompt.

Prompt:

Write a Flutter widget that creates the screen pictured in the attached design concept.

And here’s the response verbatim from GPT-4.

Response:

To create a Flutter widget that matches the design concept you’ve provided, you can follow the widget tree structure that includes a Scaffold, an AppBar (optional, if you want to include the back button as seen in the design), Image, TextFormFields for email and password, and TextButtons for the ‘Forgot Your Password?’ and ‘Log in’ functionalities. You would also likely have a link for ‘Create one’ referring to account creation.

Here is a code snippet that outlines the structure of the Flutter widget:

import 'package:flutter/material.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Login Screen',
      theme: ThemeData(
        primarySwatch: Colors.blue,
        visualDensity: VisualDensity.adaptivePlatformDensity,
      ),
      home: LoginScreen(),
    );
  }
}
class LoginScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      // AppBar is optional: Uncomment the following lines if you want to include an AppBar.
      // appBar: AppBar(
      //   leading: IconButton(
      //     icon: Icon(Icons.arrow_back),
      //     onPressed: () {
      //       // Handle back button
      //     },
      //   ),
      //   elevation: 0,
      //   backgroundColor: Colors.transparent,
      // ),
      body: Padding(
        padding: EdgeInsets.all(20.0),
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Image.asset('assets/logo.png'), // Replace with your asset image path
            SizedBox(height: 50),
            TextFormField(
              decoration: InputDecoration(
                labelText: 'Email',
                prefixIcon: Icon(Icons.email),
              ),
              keyboardType: TextInputType.emailAddress,
              // Add validation and logic
            ),
            SizedBox(height: 20),
            TextFormField(
              decoration: InputDecoration(
                labelText: 'Password',
                prefixIcon: Icon(Icons.lock),
                suffixIcon: IconButton(
                  icon: Icon(Icons.visibility_off),
                  onPressed: () {
                    // Handle password visibility
                  },
                ),
              ),
              obscureText: true,
              // Add validation and logic
            ),
            SizedBox(height: 20),
            Align(
              alignment: Alignment.centerRight,
              child: TextButton(
                child: Text('Forgot Your Password?'),
                onPressed: () {
                  // Handle forgot password
                },
              ),
            ),
            SizedBox(height: 20),
            ElevatedButton(
              child: Text('Log in'),
              onPressed: () {
                // Handle login logic
              },
              style: ElevatedButton.styleFrom(
                minimumSize: Size(double.infinity, 50), // double.infinity is the width and 50 is the height
              ),
            ),
            SizedBox(height: 20),
            TextButton(
              child: Text("Don't have account? Create one"),
              onPressed: () {
                // Handle account creation
              },
            ),
          ],
        ),
      ),
    );
  }
}

Make sure to replace 'assets/logo.png' with the path to your actual logo asset. You will also need to implement the logic for handling the actions like 'Forgot Your Password?', 'Log in', and 'Create one'. Additionally, you may want to adjust styling, spacing, and other UI elements to closely match the design you've provided.

Remember to add detailed dartdoc comments as required for your code standards.

Results in Flutter App

So, let’s plug this code into your bog standard Flutter counter app and see what comes out the other side, with the only modification to the code generated by ChatGPT being an update to the image file path as the response suggests doing.

Analysis of the Generated Code

Let’s dive into these results with some commentary.

Accuracy in UI Resemblance

Taking a look at the comparison between the original design and the output from GPT-4 above, this is not a bad start at all, especially considering that it took less than a minute. I would estimate that creating a view like this would take an experienced Flutter developer under 30 minutes to implement anyway, so not a giant task by any means, but still there is a gain to be made in productivity by using GPT-4 to at least get a start. Overall, the widget created by GPT-4 looks quite similar to the one in the Figma design.

The bad news is that the parts of the widget that the AI system didn’t quite nail are probably the ones that will be the most tedious for the human developer to go fix. This includes the outline and styling of the form fields, which can be time consuming to implement in a Flutter app because of the large number of customization options for form fields. Another example is the style of the “Log in” button which is, again, a bit time consuming for a human developer to bring into alignment with the design.

The comment made by ChatGPT saying, “if you want to include the back button as seen in the design,” is a little interesting since there is, in fact, no back button in the design.

So, in terms of accuracy to the design requirements, a human developer would either need to have some back and forth with ChatGPT to refine the output or simply fix the discrepancies manually. Either way, this refinement will cut in the time savings achieved by using ChatGPT in the first place.

Code Quality

So the visual appearance of the output from GPT-4 is not bad compared to the design but it will still require more work to match the design exactly. Let’s take a look at the quality of the code itself.

The good news is that, at least with this example, there are no outright compilation errors that would actually break a build. However, there is one significant problem with the code generated by GPT-4. This screen contains text input fields so, on a mobile device, the on-screen keyboard can be open. Because the view created by GPT-4 is not scrollable, this will result in an overflow.

As with some of the visual aspects, this is not very difficult to fix; the quick and dirty way is to just wrap the Scaffold body in a SingleChildScrollView widget. But, again, this kind of issue does cut into the time the developer saved by using ChatGPT. It could also create an opportunity for regressions if testing failed to catch this overflow problem.

There are three different types of warnings from flutter analyze . Most of the warnings are recommendations to use the const keyword to improve performance. This is something I’ve personally noticed with GPT-4; it never seems to use the const keyword. The second type of warning is “Constructors for public widgets should have a named ‘key’ parameter.” Using the key parameter allows keys to be provided to public widgets when they are useful for different purpose. Finally, there is one widget in which GPT-4 did not place the child named parameter last. The recommendation to define the child argument last improves readability.

So, on the whole, the quality of the code is decent. If this view did not involve inputs that would open the keyboard, the overflow issue would not exist. That said, other related issues might pop up, like overflows caused when the host device is set to a large text scale factor. But, again, overall, this output is not bad as a first draft of the Flutter widget for this page.

Experiment 2 — Design Reference Images to Specific Code Elements

Let’s try another experiment. In this one the focus shifts to a more detailed aspect of app development: using design reference images to generate specific code elements in Dart for Flutter applications. Here, the aim is to understand how multi-modal GPT-4 can be guided to concentrate on particular design features, such as background effects or header areas, and accurately translate them into code.

Selecting a Design Mockup

For this experiment, we are going to be mean to ChatGPT. Let’s select a highly complex background featuring complicated design elements. Furthermore, we will ask ChatGPT to build an animated background based on an image.

This is the kind of thing that, at least assuming it was created in code, would take a human a decent chunk of time to implement. So the opportunities for time savings with this example should be much greater than with the first experiment.

Generate Some Code

Let’s give this one a try too. We will use a prompt asking ChatGPT (using GPT-4) to create a widget that will match the background design in the image above and, additionally, add an animation effect. Since we are being a bit mean with a difficult task, we will give ChatGPT more information in the prompt this time.

Prompt:

Create a Flutter widget to be used as a decorative background based on the background in the attached image. As shown in the image, a randomly distributed series of dots are placed on the screen. Dots in close proximity to each other are interconnected with lines, creating a “web” effect. This decorative background should incorporate an animation in which the dots move slowly around the screen, “bouncing” off of the screen’s edges when the dots reach the bounds of the viewport.

And here’s the response verbatim from GPT-4.

Response:

Creating such a widget involves a combination of Flutter’s CustomPainter to draw the dots and lines, and AnimationController to animate the dots' movement. Here's a basic structure of the Flutter widget you could use:

import 'dart:math';
import 'dart:async';
import 'package:flutter/material.dart';

/// A point class to represent each node in the web.
class Point {
  double x, y; // The x and y coordinates of the point
  double dx, dy; // The delta movement on each axis

  Point(this.x, this.y, this.dx, this.dy);

  /// Updates the position of the point and reverses direction upon edge collision.
  void update(Size size) {
    x += dx;
    y += dy;

    // Reverse direction if the point hits the boundary
    if (x < 0 || x > size.width) dx = -dx;
    if (y < 0 || y > size.height) dy = -dy;
  }
}

/// A widget that uses the `CustomPainter` to draw the background effect.
class WebBackground extends StatefulWidget {
  const WebBackground({super.key, required this.width, required this.height});

  /// The width of the background effect.
  final double width;

  /// The height of the background effect.
  final double height;

  @override
  WebBackgroundState createState() => WebBackgroundState();
}

class WebBackgroundState extends State<WebBackground> {
  final List<Point> points = [];
  final int numberOfPoints = 50;
  final Random random = Random();

  @override
  void initState() {
    super.initState();
    for (int i = 0; i < numberOfPoints; i++) {
      points.add(
        Point(
          random.nextDouble() * widget.width,
          random.nextDouble() * widget.height,
          random.nextDouble() * 2 - 1,
          random.nextDouble() * 2 - 1,
        ),
      );
    }

    // Update the points' position over time to create the moving effect.
    Timer.periodic(const Duration(milliseconds: 50), (timer) {
      setState(() {
        for (Point point in points) {
          point.update(MediaQuery.of(context).size);
        }
      });
    });
  }

  @override
  Widget build(BuildContext context) {
    return CustomPaint(
      painter: WebPainter(points),
      child: Container(),
    );
  }
}

/// Custom painter class to draw the web-like effect using the points.
class WebPainter extends CustomPainter {
  final List<Point> points;

  WebPainter(this.points);

  @override
  void paint(Canvas canvas, Size size) {
    final paint = Paint()
      ..color = Colors.white.withOpacity(0.8)
      ..strokeCap = StrokeCap.round
      ..strokeWidth = 2.0;

    for (var point in points) {
      for (var otherPoint in points) {
        if (point != otherPoint) {
          if ((point.x - otherPoint.x).abs() < 100 && (point.y - otherPoint.y).abs() < 100) {
            canvas.drawLine(Offset(point.x, point.y), Offset(otherPoint.x, otherPoint.y), paint);
          }
        }
      }
    }

    for (var point in points) {
      canvas.drawCircle(Offset(point.x, point.y), 3.0, paint..color = Colors.white);
    }
  }

  @override
  bool shouldRepaint(WebPainter oldDelegate) => true;
}

This code provides a starting point. The WebPainter draws dots and lines, while the WebBackground stateful widget contains the animation logic. You may need to adjust the velocity, radius, and the number of dots to fit your needs. The above animation makes dots move continuously and bounce back from the edges. Ensure that you call setState to redraw the widget periodically.

Results in Flutter App

Let’s see how this code looks in a Flutter app. Since the code delivered by ChatGPT is a widget, I simply placed this widget into a Scaffold as the child of a Stack so that other design elements could be placed on top of the background effect. I also provided a background color for the Scaffold.

class _MyHomePageState extends State<MyHomePage> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: const Color(0xFF16101A),
      body: Stack(
        children: [
          WebBackground(
            width: MediaQuery.of(context).size.width,
            height: MediaQuery.of(context).size.height,
          ),
        ],
      ),
    );
  }
}

Analysis of the Generated Code

Let’s dive into these results with some commentary.

Accuracy in UI Resemblance

If you work with AI assistants like ChatGPT a lot, you can sometimes start to see the cracks in the system. Things like the code always missing const keywords as mentioned before, or repeatedly having to remind the system to add comments, or the assistant getting into a pattern of starting every single response with “Certainly!” But then you get results like this one that really bring the amazement factor back to the technology.

This result is really fairly incredible. First, ChatGPT interpreted the prompt requesting it to focus only on the background elements in the provided screenshot without any issues. And, while the colors are not quite the same, the overall design produced by ChatGPT is extremely similar to the provided example.

And then there is the quality of the animation. This would by no means be an easy task for even an experienced Flutter developer to accomplish. I would estimate that this task would occupy a healthy chunk of an afternoon to complete and ChatGPT generated this background in a few moments. While there is a bit of finecsing to do with the output from ChatGPT to make it match the design exactly, this example illustrates the potential for massive time savings by using AI assistants like ChatGPT for Flutter development.

Code Quality

The code quality in this example is again very good, if not perfect. ChatGPT again failed to use any const keywords where they were appropriate. There are also a handful of stylistic choices that I think could be improved, for example adding a few more comments for a widget of this complexity and using non-dynamic type declarations in the for-loops. However, again there are no compilation errors or major problems that would prevent the app from building.

Experiment 3 — Capturing Mood/Ambiance

The third experiment explores a more abstract yet equally fascinating aspect of app design: translating the mood or ambiance conveyed by an image into UI elements using Dart code for Flutter applications. This test aims to uncover how effectively multi-modal GPT-4 can interpret and translate abstract concepts, like mood and ambiance, into tangible UI components.

Understanding the Concept

The core idea here revolves around the challenge of converting less tangible aspects, such as the feel or mood evoked by an image, into concrete UI elements. This requires a deep understanding of how colors, shapes, and layout contribute to the overall ambiance of an application.

Selecting Images

The images selected for this experiment will be different than the previous two. Rather than using design mockups, the images in this experiment will simply be pictures that strongly evoke certain moods or themes. The key is to pick an image that distinctly conveys a particular moods, providing a clear emotional or thematic direction for the AI to follow.

So, here’s an image generated with DALL-E 3 based on the prompt, “Create a photorealistic image that strongly and distinctly conveys a dark, moody, and intimidating mood.”

Generate Some Code

The prompt for this experiment will be a little different than the others since the image provided to ChatGPT will also be different. The AI system will be asked to create a landing page for a movie that evokes a mood similar to the one conveyed by the image above. However, the prompt will not contain details about the exact emotion as part of the experiment will be to determine how well ChatGPT evaluates the desired mood from the image.

Prompt:

Create a widget for a Flutter web application used to create a landing page for a movie. The landing page should include a headline and a button to “Buy Tickets.” The landing page widget should be designed to evoke similar emotions to those conveyed by the attached image.

And here’s the response verbatim from GPT-4.

Response:

To create a Flutter widget that evokes the mood of the image you’ve attached, we’ll focus on a dark theme with a mysterious and slightly eerie ambiance, similar to what’s conveyed in the image. Here’s a conceptual outline of what the Dart code for such a widget might look like, including dartdoc comments for documentation:

import 'package:flutter/material.dart';

/// A custom Flutter widget for a movie landing page that evokes an eerie and mysterious atmosphere.
/// It features a headline and a button to purchase tickets.
class MovieLandingPage extends StatelessWidget {
  // Title of the movie to be displayed in the headline.
  final String movieTitle;
  // Callback function that is called when the 'Buy Tickets' button is pressed.
  final VoidCallback onBuyTickets;
  // Constructor to initialize the [MovieLandingPage].
  MovieLandingPage({
    Key? key,
    required this.movieTitle,
    required this.onBuyTickets,
  }) : super(key: key);
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: Colors.black, // Dark theme background color
      body: Container(
        decoration: BoxDecoration(
          image: DecorationImage(
            image: AssetImage('assets/images/movie_background.jpg'), // Placeholder for background image
            fit: BoxFit.cover,
            colorFilter: ColorFilter.mode(
              Colors.black.withOpacity(0.5),
              BlendMode.darken,
            ),
          ),
        ),
        child: Center(
          child: Column(
            mainAxisAlignment: MainAxisAlignment.center,
            children: [
              Text(
                movieTitle,
                style: TextStyle(
                  fontSize: 36,
                  fontWeight: FontWeight.bold,
                  color: Colors.white70,
                ),
              ),
              SizedBox(height: 20), // Provides spacing between the title and the button
              ElevatedButton(
                style: ElevatedButton.styleFrom(
                  primary: Colors.white24, // Button color with transparency
                ),
                onPressed: onBuyTickets,
                child: Text(
                  'Buy Tickets',
                  style: TextStyle(
                    fontSize: 20,
                    color: Colors.white,
                  ),
                ),
              ),
            ],
          ),
        ),
      ),
    );
  }
}

In this code:

• MovieLandingPage is a stateless widget because it does not manage any state changes.
• It takes movieTitle as an argument for the movie's name and onBuyTickets as a callback for when the tickets button is pressed.
• The UI uses a dark theme with a semi-transparent overlay to give a mysterious feel, similar to the attached image.
• The background image should be a dark, moody scene that fits the style of the movie. You will need to replace 'assets/images/movie_background.jpg' with the actual path to your image asset.
• The headline uses a large, bold font with a color that stands out against the dark background.
• The button is styled to fit the eerie theme with a semi-transparent white color.

To match the mood of the image, you might want to customize the widget further by adding more visual elements that align with the aesthetic of the scene in the image, like bare tree branches or fog effects.

Results in Flutter App

In this experiment, ChatGPT generated a StatelessWidget used to create a screen in a Flutter app. So, using the normal “counter” example app for a new Flutter app, the default “MyApp” home page was replaced with the one from the ChatGPT output.

As the output from ChatGPT recommended, I replaced the image placeholder with a real image.

Analysis of the Generated Code

Let’s take a look at the output from ChatGPT for this final experiment.

Accuracy in UI Resemblance

So, right off the bat, the AI system totally nailed identifying the mood conveyed in the image, which it described as “a dark theme with a mysterious and slightly eerie ambiance.” It also did a good job including the two other UI elements I described in the prompt. The rationalle for the overall design choices in this screen used to mirror the mood in the image are also quite compelling.

One could argue, however, that simply using the provided image as the background of the page in order to reflect the emotions conveyed by the image is a little like cheating. That said, again, it was successful in identifying the mood in the image, which was really the crux of this experiment.

Code Quality

The code quality from this experiment is basically the same as the previous two, with some warnings from the linter but nothing completely broken.

Concluding Thoughts

In this series of experiments leveraging ChatGPT for creating Flutter widgets, we have observed varied levels of efficacy and potential time savings. For straightforward tasks, such as drafting a basic Flutter widget for a simple login page, the AI’s output serves as a starting point. However, the nuances of design and functionality often necessitate manual adjustments by human developers. In these cases, the overall time savings might not be substantial as the developers still need to invest effort in refining and customizing the generated code to meet specific design requirements.

On the other hand, more complex tasks, such as the adaptation of a widget to create a complex animated background, exhibit the AI’s potential to deliver more significant time savings. Implementing such nuanced design features from scratch can be time-consuming. So ChatGPT’s ability to rapidly generate a rough initial implementation can be a big benefit, freeing developers to focus on higher-level aspects of the development process.

The last experiment diverged into the realm of abstraction, demonstrating ChatGPT’s capability to interpret and translate the evocative elements of a design into a corresponding digital experience. Although this was more exploratory, it showcases the versatility of GPT-4 in grasping and applying abstract design concepts, which can be especially useful in the early stages of creative processes.

As AI systems like GPT-4 continue to evolve, their role in capturing and translating design elements into tangible drafts is likely to become an invaluable asset in the developer’s toolkit, particularly in accelerating the transition from concept to prototype. It will certainly be exiting to see how these tools continue to evolve in the future.

Thanks for reading and happy Fluttering!