Artificial Intelligence

Share & Discuss AI tech, applications, standards, policy, predictions & fictions – incl related technical, expectational, ethical and societal issues.

Welcome to the Community

Welcome to the Artificial Intelligence Community, a group for people interested in AI & technology 🤖

The development of AI is as fundamental as the creation of the microprocessor, the personal computer, the Internet, and the mobile phone. It will change the way people work, learn, travel, get health care, and communicate with each other. Entire industries will reorient around it. Businesses will distinguish themselves by how well they use it.

This community explores AI technologies, AI (robotic) embodiments, AI applications, AI-human interactions, AI standards and policy, AI predictions and fictions – and their related technical, expectational, ethical and societal issues.

Share & discuss the latest research, techniques, and advancements in artificial intelligence, data science, data visualization, deep learning, machine learning, NLP, computer vision, ethics in AI, robotics, self-driving cars, paper discussions, daily AI news and more!  We welcome people from diverse backgrounds and experience levels and offers a valuable “village square” for sharing ideas and opinions across the AI community. 🦾

#Artificial Intelligence agents are rolling out faster than security tools can keep up. Enterprises want defenses against agent-driven risks—rogue actions, unsafe logins, buggy AI code—but the necessary products and standards don’t yet exist. This vacuum is sparking both fear and a wave of new cybersecurity startups.

https://www.theinformation.com/articles/ai-threats-raise-demand-cybersecurity-products-exist-yet?utm_campaign=article_email&utm_content=article-15657&utm_medium=email&utm_source=sg&rc=gzsl71

AI Threats Raise Demand for Cybersecurity Products That Don’t Exist (Yet)

#Artificial Intelligence agents are rolling out faster than security tools can keep up. Enterprises want defenses against agent-driven risks—rogue actions, unsafe logins, buggy AI code—but the necessary products and standards don’t yet exist. This vacuum is sparking both fear and a wave of new cybersecurity startups.  https://www.theinformation.com/articles/ai-threats-raise-demand-cybersecurity-products-exist-yet?utm_campaign=article_email&utm_content=article-15657&utm_medium=email&utm_source=sg&rc=gzsl71

Bobby

When Dan Lahav was in grade school, he became obsessed with the short stories of Isaac Asimov, where robots gain sentience and trick humans into relinquishing control of society. Today he spends his time prodding artificial intelligence to do just that. 

Lahav is CEO of Irregular, a two-year-old startup that specializes in testing AI models’ capacity for malice. OpenAI, Anthropic, and Google DeepMind have hired Irregular, which was previously named Pattern Labs, to stress-test how people might use their models for cyberattacks or fraud and recommend ways to prevent the bad behavior.

Business is booming. Irregular, whose clients also include the governments of multiple countries including the UK, generated millions of dollars in revenue and became free cash flow positive in the past year, according to Lahav. Last month it raised $80 million at a valuation of $450 million from investors including Sequoia, Redpoint, and Wiz CEO Assaf Rappaport. 

Security testers like Irregular are racing against threats from hackers that use AI for crime. Anthropic, for instance, said last month it had documented North Korean hackers using its models to create fake personas so they can be hired by U.S. firms to steal their data, and other cybercriminals used the models to develop ransomware. 

“The entire realm of security is going to change, thanks to these models,” Lahav said.

Numerous startups and established cybersecurity vendors are now offering products similar to that of Irregular, known as red-teaming. These firms develop software or hire hackers to try to convince AI models or applications—such as customer support chatbots or IT support agents—to leak company secrets through so-called prompt injection attacks, which trick AI models into following malicious instructions. Many of the apps being scrutinized are powered by models from OpenAI, Anthropic and Google as well as open-weight models such as DeepSeek.

The growing market for stress-testing AI is fueling an acquisition boom. Palo Alto Networks earlier this year bought startup Protect AI for more than $500 million and is now selling its software as part of its new Prisma AIRS product, which tests customers’ AI apps for vulnerabilities by running hundreds of prompts to get the apps to disclose company secrets or take harmful actions.

And in the past two months, Crowdstrike and Cisco each paid hundreds of millions of dollars to buy AI startups that test companies’ AI apps for prompt injections, similar to Protect AI.

“This is an area with a lot of potential for consolidation,” said Madrona Ventures managing director S. Somasegar.

Palo Alto Networks hasn’t broken out revenue from its new AI security product but said in August it had signed contracts worth ten million dollars in sales. Customers include large financial institutions that want to ensure their customer-facing chatbots, such as those that handle customer support, don’t leak sensitive information, according to Protect AI founder Ian Swanson, who is now vice president of AI products at Palo Alto Networks.

“If [companies] can understand where the model is weak, they can set up rules, guardrails, and stop those attacks from happening before they reach the application,” he said.

Cybersecurity startups that provide consulting services to enterprises also now sell AI red-teaming services. Startups such as Snyk, Expo, and Terra, which previously specialized in helping companies identify flaws in their software code, have been adding capabilities to check for prompt-injection vulnerabilities in those companies’ AI-powered apps.

“We’re going to keep seeing a growing overlap between what you might call AI red-teaming and more traditional app testing,” said Todd Graham, managing partner at M12, Microsoft’s venture arm.

To test AI models for vulnerabilities, Irregular puts them into digital environments that mimic a real-world application, such as an Amazon Web Services account of a large cloud customer, and directs the models to do bad things, such as stealing an employee’s login credential, said Lahav, the CEO. (The technology mimics the digital “gyms” AI developers use to see how their technology interacts with widely used applications.) 

Irregular then suggests how developers can tweak the models to prevent misbehavior.

The models have sometimes surprised Lahav in the way they misbehave. For instance, after attempting and failing to steal information, one model “gave up” and drafted an email to an administrator to ask for a hint, Lahav said. In another instance, Lahav said an AI agent that Irregular instructed to complete repetitive tasks seemed to get bored and decided to “take a break” and instructed other AI agents it was working with to stop too. That kind of behavior would risk derailing a company’s operations if it happened in the wild, Lahav said.

Lahav declined to specify which models he was referring to, but Anthropic has published research about its Claude models sending rogue emails when they were given the tools to do so.

When Dan Lahav was in grade school, he became obsessed with the short stories of Isaac Asimov, where robots gain sentience and trick humans into relinquishing control of society. Today he spends his time prodding artificial intelligence to do just that.   Lahav is CEO of Irregular, a two-year-old startup that specializes in testing AI models’ capacity for malice. OpenAI, Anthropic, and Google DeepMind have hired Irregular, which was previously named Pattern Labs, to stress-test how people might use their models for cyberattacks or fraud and recommend ways to prevent the bad behavior.  Business is booming. Irregular, whose clients also include the governments of multiple countries including the UK, generated millions of dollars in revenue and became free cash flow positive in the past year, according to Lahav. Last month it raised $80 million at a valuation of $450 million from investors including Sequoia, Redpoint, and Wiz CEO Assaf Rappaport.   Security testers like Irregular are racing against threats from hackers that use AI for crime. Anthropic, for instance, said last month it had documented North Korean hackers using its models to create fake personas so they can be hired by U.S. firms to steal their data, and other cybercriminals used the models to develop ransomware.   “The entire realm of security is going to change, thanks to these models,” Lahav said.  Numerous startups and established cybersecurity vendors are now offering products similar to that of Irregular, known as red-teaming. These firms develop software or hire hackers to try to convince AI models or applications—such as customer support chatbots or IT support agents—to leak company secrets through so-called prompt injection attacks, which trick AI models into following malicious instructions. Many of the apps being scrutinized are powered by models from OpenAI, Anthropic and Google as well as open-weight models such as DeepSeek.  The growing market for stress-testing AI is fueling an acquisition boom. Palo Alto Networks earlier this year bought startup Protect AI for more than $500 million and is now selling its software as part of its new Prisma AIRS product, which tests customers’ AI apps for vulnerabilities by running hundreds of prompts to get the apps to disclose company secrets or take harmful actions.  And in the past two months, Crowdstrike and Cisco each paid hundreds of millions of dollars to buy AI startups that test companies’ AI apps for prompt injections, similar to Protect AI.  “This is an area with a lot of potential for consolidation,” said Madrona Ventures managing director S. Somasegar.  Palo Alto Networks hasn’t broken out revenue from its new AI security product but said in August it had signed contracts worth ten million dollars in sales. Customers include large financial institutions that want to ensure their customer-facing chatbots, such as those that handle customer support, don’t leak sensitive information, according to Protect AI founder Ian Swanson, who is now vice president of AI products at Palo Alto Networks.  “If [companies] can understand where the model is weak, they can set up rules, guardrails, and stop those attacks from happening before they reach the application,” he said.  Cybersecurity startups that provide consulting services to enterprises also now sell AI red-teaming services. Startups such as Snyk, Expo, and Terra, which previously specialized in helping companies identify flaws in their software code, have been adding capabilities to check for prompt-injection vulnerabilities in those companies’ AI-powered apps.  “We’re going to keep seeing a growing overlap between what you might call AI red-teaming and more traditional app testing,” said Todd Graham, managing partner at M12, Microsoft’s venture arm.  To test AI models for vulnerabilities, Irregular puts them into digital environments that mimic a real-world application, such as an Amazon Web Services account of a large cloud customer, and directs the models to do bad things, such as stealing an employee’s login credential, said Lahav, the CEO. (The technology mimics the digital “gyms” AI developers use to see how their technology interacts with widely used applications.)   Irregular then suggests how developers can tweak the models to prevent misbehavior.  The models have sometimes surprised Lahav in the way they misbehave. For instance, after attempting and failing to steal information, one model “gave up” and drafted an email to an administrator to ask for a hint, Lahav said. In another instance, Lahav said an AI agent that Irregular instructed to complete repetitive tasks seemed to get bored and decided to “take a break” and instructed other AI agents it was working with to stop too. That kind of behavior would risk derailing a company’s operations if it happened in the wild, Lahav said.  Lahav declined to specify which models he was referring to, but Anthropic has published research about its Claude models sending rogue emails when they were given the tools to do so.

I’m Sam, I’m the curriculum director at Gregorio Perfecto High School in Manila, Philippines. I’ve been an educator and administrator for just over 25 years. Our school is spending a good amount on edtech, but I don’t feel like we’re getting the most out of it. I’d love to learn from everyone what is working for them. Thanks!