Free AI web copilot to create summaries, insights and extended knowledge, download it at here
6910
Abstract
ython_coreml_stable_diffusion.coreml_model:Loading a CoreML model through coremltools triggers compilation every time. The Swift package we provide uses precompiled Core ML models (.mlmodelc) to avoid compile-on-load.
INFO:python_coreml_stable_diffusion.coreml_model:Loading vae_decoder mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Loading ./models/Stable_Diffusion_version_CompVis_stable-diffusion-v1-4_vae_decoder.mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 5.5 seconds.
INFO:python_coreml_stable_diffusion.coreml_model:Loading safety_checker mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Loading ./models/Stable_Diffusion_version_CompVis_stable-diffusion-v1-4_safety_checker.mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 2.2 seconds.
INFO:main:Done.
INFO:main:Initializing Core ML pipe <span class="hljs-keyword">for</span> image generation
INFO:main:Stable Diffusion configured to generate 512x512 images
INFO:main:Done.
INFO:main:Beginning image generation.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [01:50<00:00, 2.17s/it]
INFO:main:Generated image has nsfw concept=False</pre></div><p id="77e7">After the program finishes, we could find the file under <code>./output</code></p><figure id="c8ab"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*4Ntawlgr5C5fwo-Ca-2shg.jpeg"><figcaption></figcaption></figure><p id="cd05">However, it takes 3–4 minutes to see result images, rather slow. To generate more variations via adjusting random “seeding” or change the “descriptive text”, all in commands. It could be inconvenient when the text is long。</p><h1 id="a3ea">Step 4. How to make it image generation easier with ML Stable Diffusion?</h1><h2 id="cda1">Let’s build a Web UI</h2><p id="0bb8"><code>gradio</code> is an interesting python library to quickly create simple stunning web UI.</p><figure id="8f88"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*s5nh1On2QyHCr8pWPOL6Wg.jpeg"><figcaption></figcaption></figure><p id="6ab7">Let’s create a <code>web.py</code> with follow content</p><div id="8584"><pre><span class="hljs-keyword">import</span> python_coreml_stable_diffusion.pipeline <span class="hljs-keyword">as</span> pipeline</pre></div><div id="c3c9"><pre><span class="hljs-keyword">import</span> gradio <span class="hljs-keyword">as</span> gr
<span class="hljs-title">from</span> diffusers <span class="hljs-keyword">import</span> StableDiffusionPipeline</pre></div><div id="da50"><pre>def init(args):
pipeline.logger.<span class="hljs-built_in">info</span>(<span class="hljs-string">"Initializing PyTorch pipe for reference configuration"</span>)
pytorch_pipe = StableDiffusionPipeline.from_pretrained(args.model_version,
<span class="hljs-attribute">use_auth_token</span>=<span class="hljs-literal">True</span>)</pre></div><div id="3c0d"><pre> user_specified_scheduler = None
<span class="hljs-keyword">if</span> <span class="hljs-built_in">args</span>.scheduler <span class="hljs-built_in">is</span> <span class="hljs-keyword">not</span> None:
user_specified_scheduler = pipeline.SCHEDULER_MAP[
<span class="hljs-built_in">args</span>.scheduler].from_config(pytorch_pipe.scheduler.config)</pre></div><div id="cb88"><pre> <span class="hljs-attr">coreml_pipe</span> = pipeline.get_coreml_pipe(pytorch_pipe=pytorch_pipe,
<span class="hljs-attr">mlpackages_dir</span>=args.i,
<span class="hljs-attr">model_version</span>=args.model_version,
<span class="hljs-attr">compute_unit</span>=args.compute_unit,
<span class="hljs-attr">scheduler_override</span>=user_specified_scheduler)
</pre></div><div id="880f"><pre> def infer(<span class="hljs-built_in">prompt</span>, steps):
pipeline.logger.info(<span class="hljs-string">"Beginning image generation."</span>)
<span class="hljs-built_in">image</span> = coreml_pipe(
<span class="hljs-built_in">prompt</span>=<span class="hljs-built_in">prompt</span>,
<span class="hljs-built_in">height</span>=coreml_pipe.<span class="hljs-built_in">height</span>,
<span class="hljs-built_in">width</span>=coreml_pipe.<span class="hljs-built_in">width</span>,
num_inference_steps=steps,
)
images = []
images.<span class="hljs-built_in">append</span>(<span class="hljs-built_in">image</span>[<span class="hljs-string">"images"</span>][<span class="hljs-number">0</span>])
<span class="hljs-built_in">return</span> images
</pre></div><div id="d83e"><pre> <span class="hljs-attr">demo</span> = gr.Blocks()</pre></div><div id="0310"><pre> with demo:
gr.Markdown(
<span class="hljs-string">"<center><h1>Core ML Stable Diffusion</h1>Run Stable Diffusion on Apple Silicon with Core ML</center>"</span>)
with gr.Group():
with gr.Box():
with gr.Row():
with gr.Column():
with gr.Row():
text = gr.Textbox(
<span class="hljs-attribute">label</span>=<span class="hljs-string">"Prompt"</span>,
<span class="hljs-attribute">lines</span>=11,
<span class="hljs-attribute">placeholder</span>=<span class="hljs-string">"Enter your prompt"</span>,
)
with gr.Row():
btn = gr.Button(<span class="hljs-string">"Generate image"</span>)
with gr.Row():
steps = gr.Slider(<span class="hljs-attribute">label</span>=<span class="hljs-string">"Steps"</span>, <span class="hljs-attribute">minimum</span>=1,
<span class="hljs-attribute">maximum</span>=50, <span class="hljs-attribute">value</span>=10, <span class="hljs-attribute">step</span>=1)
with gr.Column():
gallery = gr.Gallery(
<span class="hljs-attribute">label</span>=<span class="hljs-string">"Generated image"</span>, <span class="hljs-attribute">elem_id</span>=<span class="hljs-string">"gallery"</span>
)</pre></div><div id="6605"><pre> text.submit<span class="hljs-params">(infer, <span class="hljs-attr">inputs</span>=[text, steps], <span class="hljs-attr">outputs</span>=gallery)</span>
btn.click<span class="hljs-params">(infer, <span class="hljs-attr">inputs</span>=[text, steps], <span class="hljs-attr">outputs<
Options
/span>=gallery)</span></pre></div><div id="603f"><pre> demo.launch(<span class="hljs-attribute">debug</span>=<span class="hljs-literal">True</span>, <span class="hljs-attribute">server_name</span>=<span class="hljs-string">"0.0.0.0"</span>)
</pre></div><div id="b5d7"><pre><span class="hljs-keyword">if</span> name == <span class="hljs-string">"main"</span>:
parser = pipeline<span class="hljs-selector-class">.argparse</span><span class="hljs-selector-class">.ArgumentParser</span>()</pre></div><div id="3ee6"><pre> parser.add_argument(
<span class="hljs-string">"-i"</span>,
<span class="hljs-attribute">required</span>=<span class="hljs-literal">True</span>,
help=(<span class="hljs-string">"Path to input directory with the .mlpackage files generated by "</span>
<span class="hljs-string">"python_coreml_stable_diffusion.torch2coreml"</span>))
parser.add_argument(
<span class="hljs-string">"--model-version"</span>,
<span class="hljs-attribute">default</span>=<span class="hljs-string">"CompVis/stable-diffusion-v1-4"</span>,
help=
(<span class="hljs-string">"The pre-trained model checkpoint and configuration to restore. "</span>
<span class="hljs-string">"For available versions: https://huggingface.co/models?search=stable-diffusion"</span>
))
parser.add_argument(
<span class="hljs-string">"--compute-unit"</span>,
<span class="hljs-attribute">choices</span>=pipeline.get_available_compute_units(),
<span class="hljs-attribute">default</span>=<span class="hljs-string">"ALL"</span>,
help=(<span class="hljs-string">"The compute units to be used when executing Core ML models. "</span>
f<span class="hljs-string">"Options: {pipeline.get_available_compute_units()}"</span>))
parser.add_argument(
<span class="hljs-string">"--scheduler"</span>,
<span class="hljs-attribute">choices</span>=tuple(pipeline.SCHEDULER_MAP.keys()),
<span class="hljs-attribute">default</span>=None,
help=(<span class="hljs-string">"The scheduler to use for running the reverse diffusion process. "</span>
<span class="hljs-string">"If not specified, the default scheduler from the diffusers pipeline is utilized"</span>))</pre></div><div id="522f"><pre> <span class="hljs-variable">args</span> = <span class="hljs-variable">parser.parse_args</span>()
<span class="hljs-function"><span class="hljs-title">init</span>(<span class="hljs-variable">args</span>)</span></pre></div><p id="46e2">Save above <code>web.py</code>into <code>python_coreml_stable_diffusion</code> directory and then run</p><div id="717c"><pre>python -m python_coreml_stable_diffusion.web -i ./models --compute-unit ALL</pre></div><p id="7e7d">Here is the logs after that command</p><div id="1f63"><pre>WARNING:coremltools:Torch version 1.13.0 has not been tested with coremltools. You may run into unexpected errors. Torch 1.12.1 is the most recent version that has been tested.
INFO:python_coreml_stable_diffusion.pipeline:Initializing PyTorch pipe <span class="hljs-keyword">for</span> reference configuration
Fetching 16 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 16396.01it/s]
INFO:python_coreml_stable_diffusion.pipeline:Removed PyTorch pipe to reduce peak memory consumption
INFO:python_coreml_stable_diffusion.pipeline:Loading Core ML models <span class="hljs-keyword">in</span> memory from ./models
INFO:python_coreml_stable_diffusion.coreml_model:Loading text_encoder mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Loading ./models/Stable_Diffusion_version_CompVis_stable-diffusion-v1-4_text_encoder.mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 4.4 seconds.
INFO:python_coreml_stable_diffusion.coreml_model:Loading unet mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Loading ./models/Stable_Diffusion_version_CompVis_stable-diffusion-v1-4_unet.mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 73.5 seconds.
INFO:python_coreml_stable_diffusion.coreml_model:Loading a CoreML model through coremltools triggers compilation every time. The Swift package we provide uses precompiled Core ML models (.mlmodelc) to avoid compile-on-load.
INFO:python_coreml_stable_diffusion.coreml_model:Loading vae_decoder mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Loading ./models/Stable_Diffusion_version_CompVis_stable-diffusion-v1-4_vae_decoder.mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 6.0 seconds.
INFO:python_coreml_stable_diffusion.coreml_model:Loading safety_checker mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Loading ./models/Stable_Diffusion_version_CompVis_stable-diffusion-v1-4_safety_checker.mlpackage
INFO:python_coreml_stable_diffusion.coreml_model:Done. Took 1.9 seconds.
INFO:python_coreml_stable_diffusion.pipeline:Done.
INFO:python_coreml_stable_diffusion.pipeline:Initializing Core ML pipe <span class="hljs-keyword">for</span> image generation
INFO:python_coreml_stable_diffusion.pipeline:Stable Diffusion configured to generate 512x512 images
INFO:python_coreml_stable_diffusion.pipeline:Done.
Running on <span class="hljs-built_in">local</span> URL: http://0.0.0.0:7860</pre></div><div id="a7f4"><pre><span class="hljs-keyword">To</span> create a <span class="hljs-keyword">public</span> <span class="hljs-keyword">link</span>, <span class="hljs-built_in">set</span> <span class="hljs-string">share=True</span> <span class="hljs-keyword">in</span> <span class="hljs-string">launch()</span>.</pre></div><p id="b1df">We could see that the Web service is starting on port 7860. Let’s open our favourite browser on the address.</p><figure id="db3e"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*_HwT7zViq5n86sFosWNt2Q.jpeg"><figcaption></figcaption></figure><p id="4c1a">Let’s test it with “colourful startrails”, then click “Generate image” then wait for the generation to complete and the image will appear on the right side.</p><figure id="fb07"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*wuRf6OR_HPTEpKwshZinHw.jpeg"><figcaption></figcaption></figure><p id="470f">Now, it is much easier. We only need to adjust the text in the “prompt” and a single click would generate image, saving us from command param adjusting or digging file through the directory. In addition, the model is loaded only once unlike previously each invoke will require loading model separately, saving a lot of time too.</p><p id="9f82">If you find the guide helpful, feel free to clap and follow me. Join medium via <a href="https://medium.com/@caodanju/membership">this link</a> to access all premium articles from me and all other awesome writers here on medium.</p></article></body>
