Custom shaped JDialog needs smooth edges

The problem that I'm facing is that when I use the setWindowShape method from the com.sun.awt.AWTUtilities, is that the edges of my JDialog aren't smooth and really ugly.

I'm referring to his url where my journey started: Here. I did everything as they adviced in the article. The photo's at the beginning of the article show the non-smooth edges of the oval shaped frame.

So since there was no solution mentioned in the article about the ugly edges of non rectangular shapes, I started to google to look for a solution. This is what I come up with:

1) Anti-aliasing doesn't seems to work when I use the setWindowShape method. Since the method is used in the componentResized method, I tried to add a anti-aliasing renderinghint in the paint method of the JDialog, but that is not working.

Code:

---

fd.addComponentListener(new ComponentAdapter() {
    @Override
    public void componentResized(ComponentEvent evt) {
      raphics2D g2 = (Graphics2D) g.create();

            Double width = 500.0;
            Double height = width/2;
            Double arc = width/10;

            roundRectangle = new RoundRectangle2D.Double(0.0, 0.0, width, height, arc, arc);

            rectangleArea = new Area(roundRectangle);

            int x2Points[] = {(width.intValue() * 2) / 5, width.intValue() / 4, width.intValue() / 2};
            int y2Points[] = {height.intValue(), (height.intValue() * 7) /5, height.intValue()};

            GeneralPath polyline = new GeneralPath(GeneralPath.WIND_EVEN_ODD, x2Points.length);

            polyline.moveTo(x2Points[0], y2Points[0]);

            for (int index = 1; index < x2Points.length; index++) {
                polyline.lineTo(x2Points[index], y2Points[index]);
            }

            triangleArea = new Area(polyline);

            rectangleArea.add(triangleArea);
      AWTUtilities.setWindowShape(rectangleArea, shape);
    }
});

public void paint(Graphics g){

RenderingHints rh = new RenderingHints(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);

            g2.setRenderingHints(rh);
}

---

2) So I tried a different approach, namely drawing the shape in the paint method and use the setWindowOpaque too false. This gives me a good result, smooth edges, but this gives me other problems:
- If I want to add a panel to the JDialog, the panel and it components aren't visible, or only when i click on them.
- When I click and the JDialog appears, it seems it repaints itselfs a couple of times and it flickkers which is also not good.

Code:

---

@Override
        public void paint(Graphics g){

            AWTUtilities.setWindowOpaque(this, false);

            Graphics2D g2 = (Graphics2D) g.create();

            Double width = 500.0;
            Double height = width/2;
            Double arc = width/10;

            roundRectangle = new RoundRectangle2D.Double(0.0, 0.0, width, height, arc, arc);

            rectangleArea = new Area(roundRectangle);

            int x2Points[] = {(width.intValue() * 2) / 5, width.intValue() / 4, width.intValue() / 2};
            int y2Points[] = {height.intValue(), (height.intValue() * 7) /5, height.intValue()};

            GeneralPath polyline = new GeneralPath(GeneralPath.WIND_EVEN_ODD, x2Points.length);

            polyline.moveTo(x2Points[0], y2Points[0]);

            for (int index = 1; index < x2Points.length; index++) {
                polyline.lineTo(x2Points[index], y2Points[index]);
            }

            triangleArea = new Area(polyline);

            rectangleArea.add(triangleArea);

            RenderingHints rh = new RenderingHints(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);

            g2.setRenderingHints(rh);

            g2.setPaint(Color.MAGENTA);
            g2.fill(rectangleArea);

            g2.draw(rectangleArea);
        }

---

3) I found an article on the web, which suggests about soft clipping. The thing is that I don't know if it can be a solution, because I don't understand anything of the setComposite function of Graphics2D. Here is a link to the article: Here.


I hope anyone can help me out with this non-smooth edged JDialog. The shape is a textballoon form. I've been searching for a couple of days now, and haven't found a clear solution on the web.

Grtz

What object paint method do you provide here?

Ok, I made 2 small SSCCE's to show the problem properly:

For the first solution - 1)

Code:

---

import com.sun.awt.AWTUtilities;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.Window;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.ComponentAdapter;
import java.awt.event.ComponentEvent;
import java.awt.geom.Area;
import java.awt.geom.GeneralPath;
import java.awt.geom.RoundRectangle2D;
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.JTextField;

public class SSCCE2Test{

    private static JFrame frame;
    private static JPanel panel;
    private JButton button;

    public SSCCE2Test(){
        panel = new JPanel();
        panel.setPreferredSize(new Dimension(750, 750));

        button = new JButton("Click here to pop up a JDialog");
        button.addActionListener(new AppointmentButtonHandler());

        panel.add(button);
    }

    public void paint(Graphics g){

        RenderingHints rh = new RenderingHints(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);

        g2.setRenderingHints(rh);
    }

    private class AppointmentButtonHandler implements ActionListener
    {
        @Override
        public void actionPerformed(ActionEvent actionEvent){
            new CreateEvent();
        }
    }

    private class CreateEvent extends JDialog
    {
        private JButton addAppointmentButton, closeDialogButton;
        private JPanel mainPanel;
        private RoundRectangle2D.Double roundRectangle;
        private Area rectangleArea, triangleArea;
        private JLabel whenLabel, whatLabel;
        private JTextField whatTextField;

        public CreateEvent(){

            addComponentListener(new ComponentAdapter()
            {
                @Override
                public void componentResized(ComponentEvent evt) {

                    Double width = 500.0;
                    Double height = width/2;
                    Double arc = width/10;

                    roundRectangle = new RoundRectangle2D.Double(0.0, 0.0, width, height, arc, arc);

                    rectangleArea = new Area(roundRectangle);

                    int x2Points[] = {(width.intValue() * 2) / 5, width.intValue() / 4, width.intValue() / 2};
                    int y2Points[] = {height.intValue(), (height.intValue() * 7) /5, height.intValue()};

                    GeneralPath polyline = new GeneralPath(GeneralPath.WIND_EVEN_ODD, x2Points.length);

                    polyline.moveTo(x2Points[0], y2Points[0]);

                    for (int index = 1; index < x2Points.length; index++) {
                        polyline.lineTo(x2Points[index], y2Points[index]);
                    }

                    triangleArea = new Area(polyline);

                    rectangleArea.add(triangleArea);
                    AWTUtilities.setWindowShape((Window)evt.getComponent(), rectangleArea);
                }
            });

            mainPanel = new JPanel();
            mainPanel.setPreferredSize(new Dimension(700, 400));
            mainPanel.setBackground(Color.red);

            closeDialogButton = new JButton("Close Dialog");
            closeDialogButton.setBorder(null);
            closeDialogButton.setContentAreaFilled(false);
            whenLabel = new JLabel("Wanneer: ");
            whatLabel = new JLabel("Wat: ");
            whatLabel.setLabelFor(whatTextField);
            whatTextField = new JTextField(30);

            addAppointmentButton = new JButton("Afspraak toevoegen");

            mainPanel.add(closeDialogButton);
            mainPanel.add(whenLabel);
            mainPanel.add(whatLabel);
            mainPanel.add(whatTextField);
            mainPanel.add(addAppointmentButton);

            setContentPane(mainPanel);
            setUndecorated(true);
            pack();
            setLocationRelativeTo(null);
            setVisible(true);
        }
    }

    private static void createAndShowGUI(){

        new SSCCE2Test();

        frame = new JFrame("Custom JDialog");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.setContentPane(panel);

        frame.pack();
        frame.setLocationRelativeTo(null);
        frame.setVisible(true);
    }

    public static void main (String[] args){
        javax.swing.SwingUtilities.invokeLater(new Runnable(){
            @Override
            public void run(){
                createAndShowGUI();
            }
        });
    }
}

---

For the second solution 2)

Code:

---

import com.sun.awt.AWTUtilities;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.RenderingHints;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.geom.Area;
import java.awt.geom.GeneralPath;
import java.awt.geom.RoundRectangle2D;
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.JTextField;

public class SSCCETest{

    private static JFrame frame;
    private static JPanel panel;
    private JButton button;

    public SSCCETest(){
        panel = new JPanel();
        panel.setPreferredSize(new Dimension(750, 750));

        button = new JButton("Click here to pop up a JDialog");
        button.addActionListener(new AppointmentButtonHandler());

        panel.add(button);
    }

    private class AppointmentButtonHandler implements ActionListener
    {
        @Override
        public void actionPerformed(ActionEvent actionEvent){
            new CreateEvent();
        }
    }

    private class CreateEvent extends JDialog
    {
        private JButton addAppointmentButton, closeDialogButton;
        private JPanel mainPanel;
        private RoundRectangle2D.Double roundRectangle;
        private Area rectangleArea, triangleArea;
        private JLabel whenLabel, whatLabel;
        private JTextField whatTextField;

        @Override
        public void paint(Graphics g){

            AWTUtilities.setWindowOpaque(this, false);

            Graphics2D g2 = (Graphics2D) g.create();

            Double width = 500.0;
            Double height = width/2;
            Double arc = width/10;

            roundRectangle = new RoundRectangle2D.Double(0.0, 0.0, width, height, arc, arc);

            rectangleArea = new Area(roundRectangle);

            int x2Points[] = {(width.intValue() * 2) / 5, width.intValue() / 4, width.intValue() / 2};
            int y2Points[] = {height.intValue(), (height.intValue() * 7) /5, height.intValue()};

            GeneralPath polyline = new GeneralPath(GeneralPath.WIND_EVEN_ODD, x2Points.length);

            polyline.moveTo(x2Points[0], y2Points[0]);

            for (int index = 1; index < x2Points.length; index++) {
                polyline.lineTo(x2Points[index], y2Points[index]);
            }

            triangleArea = new Area(polyline);

            rectangleArea.add(triangleArea);

            RenderingHints rh = new RenderingHints(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);

            g2.setRenderingHints(rh);

            g2.setPaint(Color.MAGENTA);
            g2.fill(rectangleArea);

            g2.draw(rectangleArea);
        }

        public CreateEvent(){

            mainPanel = new JPanel();
            mainPanel.setPreferredSize(new Dimension(700, 400));
            mainPanel.setBackground(Color.red);

            closeDialogButton = new JButton("Close Dialog");
            closeDialogButton.setBorder(null);
            closeDialogButton.setContentAreaFilled(false);
            whenLabel = new JLabel("Wanneer: ");
            whatLabel = new JLabel("Wat: ");
            whatLabel.setLabelFor(whatTextField);
            whatTextField = new JTextField(30);

            addAppointmentButton = new JButton("Afspraak toevoegen");

            mainPanel.add(closeDialogButton);
            mainPanel.add(whenLabel);
            mainPanel.add(whatLabel);
            mainPanel.add(whatTextField);
            mainPanel.add(addAppointmentButton);

            setContentPane(mainPanel);
            setUndecorated(true);
            pack();
            setLocationRelativeTo(null);
            setVisible(true);
        }
    }

    private static void createAndShowGUI(){

            new SSCCETest();

            frame = new JFrame("Custom JDialog");
            frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
            frame.setContentPane(panel);

            frame.pack();
            frame.setLocationRelativeTo(null);
            frame.setVisible(true);
        }

        public static void main (String[] args){
            javax.swing.SwingUtilities.invokeLater(new Runnable(){
                @Override
                public void run(){
                    createAndShowGUI();
                }
            });
        }
    }

---

I hope this helps

Several days have past and it seems no one has found a solution yet. Is it just impossible?